Thomas Letan's Blog

What happened in February 2026?

March 8, 2026

What happened in February 2026?

Time flies. February was a productive month, and I got many things done—with the help of my coding assistant , that is. I haven’t joined the “I haven’t written a single line of code for quite some time” flock just yet, but I am close. At the same time, I feel like I’m still locked in beginner mode. Like I have still to really tap into what agentic tools like Claude Code have to offer.

For now, the tools may have changed, but the journeys feel somewhat the same as they used to a few months back. It’s a strange feeling, and I am still trying to wrap my head around it.

Yet Another Claude Code Wrapper

In my previous retrospective, I mentioned CeCe and Vee. The former is a plugin for turning Anthropic’s model into a “modal” agent that I had vibe “prompted” with Claude itself. The latter was a second take at the exact same concept.

A month later, Vee has evolved quite a bit, to a point where the “modal” aspect of the project lost the front seat. More precisely, Vee has become a session multiplexer for Claude Code, with built-in support for spawning it in ephemeral containers with --dangerously-skip-permissions enabled. It is also a playground for me to experiment with the agent. For instance, I built a feedback system where I can point it to the good and bad things it’s doing—my feedbacks get saved in a SQLite database and appended in the system prompt during the next sessions. I’ve also tried to set up a knowledge base with embeddings. Both approaches have their benefits, but I don’t think I have clear evidence that they can be a game changer.

You may remember Vee started out of my frustration to losing control of CeCe’s internals. Well, Claude Code took over Vee’s codebase quite quickly as well. But this time, the output was a lot more satisfying. This time, I leaned into playing the role of a Tech Lead and a QA tester. Seeing the project taking shape and the tool I wanted becoming a reality that quickly has clear upsides. I also find it interesting that I have both a fairly good understanding of how the software works overall—I called most of the architectural choices—and very little insight on how the code itself is written. It reminds me of $WORK, where I am involved in some projects without being a direct IC.

I Built Myself a Cloudlab

In the first half of the month, a colleague of mine was tasked with building “staging environments à la demande” for the engineers working on $WORK’s Next Big Thing™. Their answer was a managed Kubernetes cluster and a bunch of Helm charts—and a small CLI tool to deploy the latter onto the former. A few hours later, I started building myself a k3s cluster. It’ll be a fun way to get familiar with the stack, I thought. What emerged almost resembles a managed cluster of sorts. I called it elsa , and it now hosts this very website along with a few other self-hosted services I had under my radar for a while .

Note

Yes, GitHub Pages and the like remain a better, cheaper, superior even way to host my website. But where is the fun in that? 😁

This may come as a surprise, considering I had migrated my website to a new setup called tinkerbell barely a month ago. Well, fret not. The little VPS sure has been retired early, but its legacy has not been lost. elsa was built upon the very same tools (Terraform, CoreOS, and Ignition) and promise I got hooked on a month ago: that there is freedom in designing a system whose destruction is a non-event. If anything, working on elsa has only deepened my understanding—and my appreciation—of what Fedora has been pushing for with CoreOS.

I really, really want to publish an article about elsa. I yet have to find my angle, though. Funny story, the first couple of paragraphs of this section were initially intended to be its introduction, until I grew dissatisfied with them and decided to repurpose them. Which means I need to restart from scratch. 😆

Anyway, as I was making good progress on elsa, I ran into some good opportunities to contribute to some open source software. I opened two bugfix PRs over the course of my journey: one for the official BetterStack’s Terraform provider , and the other for external-dns’ Vultr webhook sidecar . That was pretty cool! I want to do that more often.

We are almost a third into March already, and the month is clearly following the trend set by its predecessor. As such, I expect to have plenty to write about in a month’s time, meaning I should be able to keep up with publishing these retrospective pieces for a while. At least, I hope so!

Teaching Firefox About slack:// URLs

February 19, 2026

Teaching Firefox About `slack://` URLs

releases

I’ve been using Slack at $WORK for the past 5 (!) years. I don't know why, but it looks like Firefox does not know what to do with the slack:// URLs out of the boxTry logging in Slack when your browser does not let Slack know authentication concluded successfully. The worst thing? Sometimes it works, sometimes it doesn’t. . It could be an issue on my side—wouldn’t be the first time!—but in case you are suffering from the same issue, I have found that I can provision my laptop with a so-called Firefox policy to fix this annoyance.

Caution

Firefox only supports a single policies.json file. If you happen to already have one installed at /usr/lib/firefox/distribution/policies.json, you will need to merge the policies manually.

{
  "policies": {
    "Handlers": {
      "schemes": {
        "slack": {
          "action": "useHelperApp",
          "ask": false,
          "path": "/usr/lib/firefox-slack-policy/open-slack"
        }
      }
    }
  }
}

This will only work if you also install the companion script open-slack.

#!/bin/sh
exec xdg-open "$@"

These two files together tell Firefox to use xdg-open in place of gio open. If I understand correctly, the latter needs an explicit entry in the mimeapps.list file to select an application to handle mimetypes it is not familiar with (like x-scheme-handler/slack) even if you have a .desktop file with the appropriate MimeType declaration. I have no idea why, especially considering that mimeinfo.cache correctly resolves x-scheme-handler/slack on my machine.

Anyway, now at least I can log in Slack without issue. I made a small AUR package of it, so if you are an ArchLinux user affected by this bug and without a pre-existing Firefox entreprise policy, feel free to install firefox-slack-policy and call it a day.

What Happened in January 2026?

January 30, 2026

What Happened in January 2026?

meta vibecoding

This is my first retrospective in quite a while—I have yet to make writing these logs a real habit of mine. That being the case, so much happened this month that this article felt like the obvious thing to do.

`tinkerbell`’s First Month

On January 5th, I published my account of migrating my website to a completely new setup. Not only am I very proud of this article in and of itself, I am also quite happy that it has sparked discussions in a few places such as lobste.rs or the orange website . It has also been shared in other places like the DevOps’ish newsletter I have also created a GitHub repository to host tinkerbell configuration, and it is quietly farming stars as we speak. .

I don’t write articles with the expectation that they reach a large audience, but I will admit I had some ambitions for this one. It’s very fulfilling to know my experiment report of sorts has caught the attention of my peers, if only a little.

At first, I thought I would spend January experimenting with my new playground. I have a handful of services I want to deploy. Little did I know I would be sidetracked sharply before I had a chance to get to it.

Meeting with Claude Code

Somehow, I have managed to go through 2025 while staying afar from the code agent hype. I gave “vibecoding” a try in May by chatting with ChatGPT and Gemini while yak shaving my way towards transcribing YouTube videos in OCaml. Even I realized back then that the hype had already moved on from chat UI towards agents.

Then, the craziest thing happened. I went on holidays on December 23rd, 2025. Two weeks later, as I was going back to $WORK, Claude Code and its ilk suddenly felt inevitable. It looks like agentic workflows got very good in a matter of a few weeks, up to a point where there is a real opportunity cost in ignoring them altogether. And as I am about to take out a mortgage, it feels like a bad time to take the risk of falling out of relevance.

Looking back, I think it started with a simple, genuine suggestion—we were discussing a fairly ambitious clean-up of our test-suite at $WORK when someone mentioned Claude Code was quite relevant for this kind of tedious, boilerplate-heavy task. This resonated with all the success stories I was suddenly exposed to. Before I could realize what was happening, I was caught up in an intense introspective journey.

Part of my answer to the angst of LLMs being on the verge of reshaping the way I work is CeCe CeCe being a nickname for Claude Code (CC). .

CeCe is a Claude Code plugin ~~I have been working~~ Claude Code and I have been working on for the better part of the month, and it has become my sandbox to experiment with and understand agentic workflows. Configuring Claude Code in a very opinionated way felt only natural—that’s the only starting point I know. I am glad I did that, it enabled me to get a better understanding of how Claude Code actually works.

Ironically, since CeCe has mostly been written by CeCe itself and because it grew quite fast as a result, I do not really feel confident in modifying it myself 😆. I am now trying to rewrite it on my own, with more structure and “intention.” We will see how it goes.

New RSS Feeds

On a final note, I am glad to say that this website now features some new RSS feeds that you may be interested inThat’s actually the very first task I entrusted to Claude Code. Suffice it to say, it didn't blink, and navigated through my quite unusual setup like a champ. : one per tags, and one per series. I have been meaning to implement them since I’ve added my website to a certain aggregator that now advertises every article I publish that cites one particular programming language.

I am glad to know that camel aficionados learnt about my tinkerbell setup simply because I drew a parallel between infrastructure as code and functional programming. Maybe they don’t share my enthusiasm, though. With this change, I am one PR away from limiting this aggregator’s scope to simply the articles featuring the appropriate tag. I think they will find it useful. Maybe you will too!

How I Want to Use LLMs in 2026

January 25, 2026

How I Want to Use LLMs in 2026

opinions vibecoding

I would like to thank Xavier Van de Woestyne for his feedback and careful review.

Agentic tools are here, and they are here to stay. I don’t think it is an overstatement to say that LLMs are completely reshaping our day-to-day life. Even if mass adoption has yet to happenWe are seeing more and more public statements from key figures of our industry like Satya Nadella or Jensen Huang trying to shed positive light on AI and pushing for more people to embrace it—probably because they don’t see the increase in active users they were hoping for. , the consequences are already here. In the software engineering industry, I already witness how they come with expectations whether we want to use them or not.

In 2025, I stayed away from the agentic hype. This month, I was made acutely aware of how transformative they can be. I don’t think there is a way back from there.

As we jump into 2026 headfirst, I consequently find myself at a crossroads. I will integrate LLMs in my workflow so that I can get the most out of what they can offer, but I want to do so consciously. Hence this article, whose tone and underlying motivation are different from my other pieces. I want to set a bar, to form a contract of sorts between me and my future self.

I want to be transparent about who—or what—produced the work I am exposing to others. I am an engineer; I can expect to generate tons of code and technical documentation using (meticulously prompted) agents. I am also publishing content online, like on this very website. The last thing I want is to trick people into thinking that I wrote something that was actually generated by a tool. Or, even worse, that they end up convinced something I genuinely wrote “the old way” has been generated.

This does not mean generated content is without value. Nor do I think there is a clear, objective line to draw between what are actually two ends of a spectrum. After all, I’ve been using ChatGPT to polish my articles for a year now, and I haven’t advertised it in the past. Still, agentic tools have become good enough: I will be in a position to describe complex tasks, fully delegate their implementation, and be confident enough to publish the result. I believe it is fair for my fellow humans to be aware of that fact when they read or review the result. Only then will they be able to calibrate their own expectations in light of this information.

As a concrete example, I have started to set up dedicated accounts for Claude Code . I am trying to come up with a reliable way to let it take over the execution of well-scoped tasks, up to responding to reviewers’ feedback on its own. That’s not an end I want to pursue unless I can be transparent about it.

I want to be deliberate about when I use or don’t use LLMs. And I have a surprising number of reasons why.

On a personal level, there are skills I don’t want to lose, craft I still wish to improve. I’m fine with never having to write a bug report directly again, but do I want to forgo authorship of my blog’s articles, for instance? Clearly not.

Besides, we are still grimly on track when it comes to climate change. Even after accepting that individual behaviors have a much less impact than what we’d like, is it really reasonable to have hours of back-and-forth with an agent every day from now on? A lot has been said and written about the impact of LLMs in that regard. I am under the impression that we recently read stories that are tragically similar to what was published about Bitcoin a few years backI had a risk management training once, and something the instructor said stuck with me: when something becomes safer, we humans tend to adapt our behavior to take more risks. Are we doing the same thing with climate change? When we manage to reduce our environmental impact, do we collectively interpret that news as a blank check to find new ways to consume more energy? .

I think one answer to this is to refuse to make the LLM the default, obvious choice. Using an agent should have weight to it. There will be times when an agent will be an enabler—achieving something outside of my immediate reachBecause it would take me too much time, because it would be extremely tedious and error-prone, or for an infinite number of valid reasons. . And there will be times when I will want to use it to write a trivial patch that I could come up with myself in a matter of minutes. I want to cultivate the discipline to make the distinction between the former and the latter, so that I avoid falling into wasteful habits.

I want to be respectful of the people who will be confronted with my use of LLMs. The matter is too complicated to approach any other way. What is acceptable for some feels like an attack on others.

Yes, agents can be powerful accelerators. No, adopting them is neither easy nor the obvious, right thing to do. We are seeing too many people being hurt by the behaviors enabled by agents’ capabilities: open-source maintainers closing their projects from external contributions, junior engineers struggling like never before to find jobs, artists witnessing in real time their art being regurgitated by models trained on their portfolios. The list goes on.

I cannot commit to never using generated art, or to close my eyes to what LLMs can bring me. But I want to be aware—and to care—about the consequences of my choices, as well as the broader context behind them. And sometimes, that needs to mean renouncing the convenience that a technology can bring, even if for just a moment.

Only time will tell how I end up using LLMs, and whether the principles I claim today as my own in this article will stand the test of time. I am rather curious to reread this piece in a year—I hope it will prompt (ah!) me to write a retrospective.

I Cannot SSH Into My Server Anymore (And That’s Fine)

January 5, 2026

I Cannot SSH Into My Server Anymore (And That’s Fine)

coreos docker meta self-hosting terraform vultr

I would like to thank Yann Régis-Gianas, Sylvain Ribstein and Paul Laforgue for their feedback and careful review.

To kick off 2026, I had clear objectives in mind: decommissioning moana, my trusty $100+/month VPS, and setting up tinkerbell, its far less costly successor.

On the one hand, I have been using moana to self-host a number of services, and it was very handy to know that I had always a go-to place to experiment with whatever caught my interest. On the other hand, $100/month is obviously a lot of money, and looking back at how I used it in 2025, it was not particularly well spent. It was time to downsize.

Now that tinkerbell is up and running, I cannot even SSH into it. In fact, nothing can.

There is no need. To update one of the services it hosts, I push a new container image to the appropriate registry with the correct tag. tinkerbell will fetch and deploy it. All on its own.

In this article, I walk through the journey that led me to the smoke and mirrors behind this magic trick: Fedora CoreOS , Ignition and Podman Quadlets in the main roles, with Terraform as an essential supporting character. This stack checks all the boxes I care about.

Note

For interested readers, I have published tinkerbell’s full setup on GitHub. This article reads as an experiment log, and if you are only interested in the final result, you should definitely have a look.

Container-Centric, Declarative, and Low-Maintenance

Going into this, I knew I didn’t want to reproduce moana’s setup—it was fully manualIn the end, I never took the time to publish a write-up about it, so in a nutshell: everything relied on a small script that enabled me to create interconnected nspawn containers on the spot. and I no longer have the time or the motivation to fiddle with the internals of a server. Instead, I wanted to embrace the principles my DevOps colleagues had taught me over the past two years.

My initial idea was to start with this very website, since it was the only service deployed on moana that I really wanted to keep. Since I had written a container image for this website, I just had to look for the most straightforward and future-proof way to deploy it in production™— something I could later extend to deploy more cool projects, if I ever wanted toIf my goal was limited to host a static website, this whole setup would arguably be both overengineered and counterproductive. tinkerbell was to become my little foothold in the Internet, though. My “cloud homelabs,” of sorts. .

Docker Compose alone wasn’t a good fit. I like compose files, but one needs to provision and manage a VM to host them. Ansible can provision VMs, but that road comes with its own set of struggles. Writing good playbooks has always felt surprisingly difficult to me. In particular, a good playbook is supposed to handle two very different scenarios—provisioning a brand new machine, and updating a pre-existing deployment—and I have found it particularly challenging to ensure that both paths reliably produce the same result.

Kubernetes was very appealing on paper. I have seen engineers turn compose files into Helm charts and be done with it. If I could do the same thing, wouldn’t that be bliss? Unfortunately, Kubernetes is a notoriously complex stack, resulting from compromises made to address challenges I simply don’t face. Managed clusters could make things easier, but they aren’t cheap. That would defeat the initial motivation behind retiring moana.

CoreOS , being an operating system specifically built to run containers, obviously stood out. That said, I had very little intuition on how it could work in practice. So I started digging. I learned about Ignition first. Its purpose is to provision a VM exactly once, at first boot. If you need to change something afterwards, you throw away your VM and create a new one. This may seem counter-intuitive, but since it eliminates the main reason I was looking for an alternative to Ansible, I was hookedCoreOS and Ignition enable me to think about virtual machines the same way OCaml or Haskell trained me to think about data: as immutable values. .

I found out how to use systemd unit files to start containers via podman CLI commands. That was way too cumbersome, so I pushed on for a way to orchestrate containers à la Docker Compose. That’s when I discovered Podman Quadlets and auto-updates .

With that, everything clicked. I knew what I wanted to do, and I was very excited about it.

Assembling `tinkerbell`

For more than a year now, my website has been served from RAM by a standalone, static binary built in OCaml, with TLS termination handled by Nginx and certbot’s certificates renewal performed by yours trulyI didn’t lie when I said moana’s setup was indeed manual. . I didn’t have any reason to fundamentally change this architecture. I was simply looking for a way to automate their deployment.

Container-Centric, …

The logical thing to do was to have tinkerbell run two containers:

The reverse proxy: I had been firmly on Team Nginx for years now, but when I heard Caddy could “automatically obtain and renew TLS certificates,” I was sold on giving it a try.
The website itself: Static binaries can be wrapped inside a container with close to zero overhead using the scratch base image, so I did just that. I published it to a free-plan, public registry hosted on Vultr that I created for the occasionWhich means getting an offline copy of this website is now as simple as calling docker pull ams.vultrcr.com/lthms/www/soap.coffee:live. .

Nothing beats a straightforward architecture

Nothing fancy or unexpected here, which made it a good target for a first deployment. It was time to open Neovim to write some YAML.

Declarative, …

At this point, the architecture was clear. The next step was to turn it into something a machine could execute. To that end, I needed two things: first an Ignition configuration, then a CoreOS VM to run it.

The Proof of Concept

Ignition configurations (.ign) are JSON files primarily intended to be consumed by machines. They are produced from YAML files using a tool called Butane . For instance, here is the first Butane configuration file I ended up writing. It provisions a CoreOS VM by creating a new user (lthms), along with a .ssh/authorized_keys file allowing me to SSH into the VMI didn’t know at the time that I would deliberately remove these lines from the final Butane file. .

variant: fcos
version: 1.5.0
passwd:
  users:
    - name: lthms
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKajIx3VWRjhqIrza4ZnVnnI1g2q6NfMfMOcnSciP1Ws lthms@vanellope

What’s important to keep in mind is that Ignition runs exactly once, at first boot. Then it is never used again. This single fact has far-reaching consequences, and is the reason why any meaningful change implies replacing the machine, not modifying it.

Before going any further, I wanted to understand how the actual deployment was going to work. I generated the Ignition configuration file.

butane main.bu > main.ign

Then, I decided to investigate how to define the Vultr VM in Terraform. The resulting configuration is twofold. First, we need to configure Terraform to be able to interact with the Vultr API, using the Vultr provider . Second, I needed to create the VM For discovering what values to put in most fields, vultr-cli is pretty convenient. Kudos to the Vultr team for making it in the first place. and feed it the Ignition configuration.

resource "vultr_instance" "tinkerbell" {
  region = "cdg"
  plan = "vc2-1c-1gb"
  os_id = "391"

  label = "tinkerbell"
  hostname = "tinkerbell"

  user_data = file("main.ign")
}

And that was it. I invoked terraform apply, waited for a little while, then SSHed into the newly created VM with my lthms user. Sure enough, the tinkerbell VM was now listed in the Vultr web interface. I explored for a little while, then called terraform destroy and rejoiced when everything worked as expected.

The MVP

At this point, I was basically done with Terraform, and I just needed to write the Butane configuration that would bring my containers to life. As I mentioned earlier, the first approach I tried was to define a systemd service responsible for invoking podman.

systemd:
  units:
    - name: soap.coffee.service
      enabled: true
      contents: |
        [Unit]
        Description=Web Service
        After=network-online.target
        Wants=network-online.target

        [Service]
        ExecStart=/usr/bin/podman run \
          --name soap.coffee \
          -p 8901:8901 \
          --restart=always \
          ams.vultrcr.com/lthms/www/soap.coffee:latest
        ExecStop=/usr/bin/podman stop soap.coffee

        [Install]
        WantedBy=multi-user.target

Adding this entry in my Butane configuration and redeploying tinkerbell got me exactly what I wanted. My website was up and running. For the sake of getting something working first, I added the necessary configuration for Caddy (the container and the provisioning of its configuration file), redeployed tinkerbell again, only to realize I also needed to create a network so that the two containers could talk together. After half an hour or so, I got everything working, but was left with a sour taste in my mouth.

This would simply not do. I wasn’t defining anything, I was writing a shell script in the most cumbersome way possible.

Then, I remembered my initial train of thought and started to search for a way to have Docker Compose work on CoreOSWell, Podman Compose, I guess. . That is when I discovered Quadlet, whose initial repository does a good job justifying its existence This repository is now archived, since Quadlet has got merged upstream. . In particular,

With quadlet, you describe how to run a container in a format that is very similar to regular systemd config files. From these actual systemd configurations are automatically generated (using systemd generators ).

To give a concrete example, here is the .container file I wrote for my website server.

[Container]
ContainerName=soap.coffee
Image=ams.vultrcr.com/lthms/www/soap.coffee:live

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

I wasn’t wasting my time teaching systemd how to start containers anymore. I was now declaring what should exist, so that systemd—repurposed for the occasion as a container orchestrator—could take care of the rest.

Tip

If your containers are basically ignored by systemd, be smarter than me. Do not try to blindly change your .container files and redeploy your VM in a very painful and frustrating loop. Simply ask systemd for the generator logs.

sudo journalctl -b | grep -i quadlet

I excitedly turned caddy.service into caddy.container, redeployed tinkerbell, ran into the exact same issue I had encountered before and discovered the easiest way for two Quadlet-defined containers to talk to each other was to introduce a pod . Unlike Docker Compose which uses DNS over a bridge network, a pod shares the network namespace, allowing containers to communicate over localhost.

To define a pod, one needs to create a .pod file, and to reference it in their .container files using the PodName= configuration option. A “few” redeployments later, I got everything working again, and I was ready to call it a day.

And with that, tinkerbell was basically ready.

Caution

I’ve later learned that restarting a container that is part of a pod will have the (to me, unexpected) side-effect to restart all the other containers of that pod.

And Low-Maintenance

Now, the end of the previous section might have given you pause.

Even a static website like this one isn’t completely “stateless.” Not only does Caddy require a configuration file to do anything meaningful, but it is also a stateful application as it manages TLS certificates over time. Besides, I do publish technical write-ups from time to timeTwo in 2025, that’s true. But 2026 is only starting, you never know what might come! .

Was I really at peace with having to destroy and redeploy tinkerbell every time I need to change anything on my website?

On the one hand, yes. I believe I could live with that. I modify my website only a handful of times even in good months, I think my audience could survive with a minute of downtime before being allowed to read my latest pieces. It may be an unpopular opinion, but considering my actual use case, it was good enough. Even the fact that I do not store the TLS certificates obtained by Caddy anywhere persistent should not be an issue. I mean, Let’s Encrypt has fairly generous weekly issuance limits per domain How do I know that? Well... I might have hit the limit while hacking my way to a working setup. . I should be fine.

On the other hand, the setup was starting to grow on me, and I have other use cases in mind that could be a good fit for it. So I started researching again, this time to understand how a deployment philosophy so focused on immutability was managing what seemed to be conflicting requirements.

I went down other rabbit holes, looking for answers. The discovery that stood out the most to me—to the point where it became the hook of this article—was Podman auto-updates.

To deploy a new version of a containerized application, you pull the new image and restart the container. When you commit to this pattern, why should you be the one performing this action? Instead, your VM can regularly check registries for new images, and update the required containers when necessary.

In practice, Podman made this approach trivial to put in place. I just needed to label my containers with io.containers.autoupdate set to registry, enable the podman-auto-update timerBy default, the timer is triggered once a day, which felt unnecessarily long, so I decided to make it hourly instead. , and that was it. Now, every time I update the tag www/soap.coffee:live to point to a newer version of my image, my website is updated within the hour.

And that is when the final piece clicked. At this point, publishing an image becomes the only deployment step. I didn’t need SSH anymore.

The Road Ahead

tinkerbell has been running for a few days now, and I am quite pleased with the system I have put in place. In retrospect, none of this is particularly novel. It feels more like I am converging toward a set of practices the industry has been gravitating toward for years.

A man looking at the “CoreOS & Quadlets” butterfly and wondering whether he’s looking at Infrastructure as Code. I’m not entirely sure of the answer.

The journey is far from being over, though. tinkerbell is up and running, and it served you this HTML page just fine, but the moment I put SSH out of the picture, it became a black box. Aside from some hardware metrics kindly provided by the Vultr dashboard, I have no real visibility into what’s going on inside. That is fine for now, but it is not a place I want to stay in forever. I plan to spend a few more weekends building an observability stackOh, and maybe I will move these TLS certificates in a block storage or something. That could be a good idea. . That will come in handy when things go wrong—as they inevitably do. I would rather have the means to understand failures than guess my way around them.

Did I ever mention I am an enthusiastic Opentelemetry convert?

Peer-Programming in Modern OCaml with ChatGPT and Gemini

June 2, 2025

Peer-Programming in Modern OCaml with ChatGPT and Gemini

ocaml vibecoding

It is June 2025, and LLMs are everywhere and do everything now. I have never been a diligent adopter of them myself. The past few months, I started to feel a bit “left out,” though. Colleagues and friends are starting to integrate LLM-powered tools into their personal toolkit, with notable successes.

Early May, I decided to challenge myself to implement a simple tool to generate a summary from YouTube videos using Vosk for speech recognition and Ollama for generating summaries using LLMs running locally. I could hit two birds with one stone—experimenting with LLMs to write and power software.

I decided to implement as much as possible in OCaml, for two main reasons. Firstly, this is the main language I use at $WORK. I wanted to get a sense of how LLMs could help with the software stack I used 7+ hours a day. Secondly, it was a good opportunity to catch-up with the OCaml 5 ecosystem (Eio in particular).

This write-up is a sort of dev log of this exercise. Its main focus is not to explain in depth the code I ended up writing, but rather to recollect on my wins and losses in adding LLMs in my developer toolkit.

TL;DR

In this article, I am using “Tip” blocks to highlight my key findings and lessons learned. That being said, for readers in a hurry, here’s how ChatGPT summarizes these blocks.

Prompting is a skill that improves through trial and error—many failed prompts help build intuition.
LLMs may suggest non-existent functions; using LSP tools helps identify these quickly.
Standard formats like WAV lead to more accurate LLM outputs.
LLMs without session memory tend to repeat mistakes; shared context is important.
Structuring commit message prompts (e.g., What / Why / How) produces consistently good results.
LLMs struggle with libraries like Eio, possibly due to name ambiguity or unstable APIs.
Providing project-specific context (e.g., via direnv) is likely to help reduce repeated hallucinations.
Prompting LLMs for MR descriptions or commits can eliminate empty submissions and speed up review.

You should still definitely read the full piece, though. I don’t think my prompt was particularly good 🤫.

Editor Integration

My first task was to grant myself the ability to leverage LLMs from my editor. I had been using the web chat of ChatGPT for a while, but it now felt antiquated since I had seen a freshly hired coworker get ChatGPT to generate for themselves a dozen tests directly from VS Code.

I have returned to Neovim for a few years, and I am not ready to migrate to VS Code. I would have been surprised if the Vim/Neovim communities wouldn’t have a viable plugin for me, though.

I asked both ChatGPT and Gemini to find my candidates, but the plugins they suggested seemed unmaintained, often outdated.

In the end, I found CodeCompanion.nvim by myself, through a good old Google research. I asked ChatGPT why it hadn’t suggested it to me, and it seems like my prompt were biased. By asking for “a Neovim ChatGPT plugin” or “a plugin to integrate Gemini to Neovim,” I had unnecessarily narrowed the LLM scope.

Tip

I guess one does not become a prompt engineer in a day. This is actually one of the reasons I want to use LLMs more seriously. To build myself intuitions of which prompts work and which don’t. After this project, I have mostly uncovered a bunch of the latter category 😅.

@yurug had told me he was impressed by Gemini Pro, so I decided to make it the default adapter for the CodeCompanionChat command. I tried to make Gemini Pro the default model for this adapter, it was challenging and LLMs weren’t able to help. When I finally found the correct setup option, it turns out I hadn’t generated a token allowing me to use Pro.

Well. That gave me the opportunity to benchmark Gemini Flash, then.

Speech Recognition with Vosk

ChatGPT suggested Vosk as a way to get a transcript of an audio file, so it was also a good opportunity to write bindings (something I had dodged for a long time for no particular reason).

As of June 2025, there is no OCaml bindings for the Vosk API , so my first task was to write my own as part of a project soberly called ocaml-vosk .

Gemini Flash was able to help me understand how ctypes and ctypes.foreign works. This was my first experience interacting with an LLM from my Neovim window, and it was pretty convincing. It gave me the opportunity to learn that one can declare opaque types in OCaml (not just via mli files). It makes sense, but it was news to me.

Then, Gemini suggested me to use EIO’s Switch to deal with automatic memory management (in place of Gc.finalise). It was the first time I heard about it, and the fact that I learned their existence from the perspective of resource management (not fiber management) was a good accident.

The first point of friction came when I started build a high-level interface for my Vosk bindings. More specifically, given a Cstruct.t value, how do I get a pointer and a length? It turns out that while both ChatGPT and Gemini Pro know how to do so, Gemini Flash hallucinates every step of the way.

The solution is actually pretty straightforward.

let ptr =
  Ctypes.bigarray_start
    Ctypes.array1
    (Cstruct.to_bigarray buffer)
in
let len = buffer.Cstruct.len in

Gemini Flash kept suggesting I use Ctypes.ptr_add instead, though. Don’t search for it, it does not existWhile reviewing this article, ChatGPT gently hinted that while ptr_add does not exist, Ctypes.(+@) does. . When I suggested Cstruct.to_bigarray, it warned me about the fact that this call would create a copy of the underlying buffer. ChatGPT and Gemini Pro disagreed, and I could convince myself that they were right by looking at the code. Interestingly, I was also able to convince Gemini Flash it was wrong by copy/pasting the relevant code snippet.

Tip

Having an LLM suggesting you to use a function which does not exist is very frustrating. Especially if it happens several times in a row—it recognizes its mistake and proposes an alternative that is as nonexistant as the first one. At least, with LSP it is pretty straightforward to know when it happens.

Using Vosk is one thing, but then I couldn’t find any OCaml package to read audio files compatible with Vosk expectations. Implementing what I needed in OCaml gave me more opportunities to learn about EIO, but most importantly, it showed how having a chat with an LLM directly from my editor was convenient. I was able to learn about WAV files, RIFF header and subchunks and PCB 16-bit mono audio data without leaving Neovim. And by giving Gemini access to my buffer, I troubleshot most of my issues fairly quickly (except when they were EIO-specific—more on that later).

Tip

For widespread encoding like WAV files, LLMs shine particularly bright.

In the end, EIO-specific code put aside, this task was roughly solved by (1) writing bindings for the few functions of the Vosk API I needed, and (2) translating C examples provided by Gemini into good-looking OCamlIt’s a little out of scope for this article, but I discovered when writing the high-level API for Vosk that Switches are very easy to misuse. It is as simple as (incorrectly) turning an eager function consuming a buffer into a Seq-based alternative, while forgetting the use of Switch.run on top of the function. .

Witnessing my example program outputting the transcript of audio files as it was processing them felt pretty good, and I was soon ready to tackle the second part of this project: prompting a LLM to summarize it.

Prompting Local LLMs with Ollama

Similarly to Vosk, there is no on the shelf package available to use Ollama from an OCaml program. As a consequence, I created a second repository (ocaml-ollama if you can believe it).

How It Started

Turns out, you don’t use Ollama the same way you use Vosk. The latter is a C library that you can call from your binary, the former actually uses a client/server architecture. I asked LLMs what was the best solution for performing HTTP requests with Eio, and cohttp-eio came back as a good candidate. I’m already familiar with cohttp, since we are using it at $WORK, but it’s actually a transitive dependency (of a framework called resto ).

I am actually a little frustrated with resto, so I welcomed the opportunity to familiar myself a little more with cohttp directly. I quickly implemented the helper fetching the list of models available from a given Ollama instance.

Then, I got myself side tracked.

More LLMs Lies

Persistent HTTP connections are a pet peeve of mine. Establishing a TCP connection, negotiating TLS encryption, all of that takes time—creating a new socket for each request a daemon really frustrates me as a result.

So I asked.

Does cohttp-eio reuses already established connections when performing two requests on the same host?

ChatGPT 4o. Gemini 2.5 Flash. Gemini 2.5 Pro. They all assured me it was the case, as long as I was careful and reused the same Cohttp_eio.Client.t instance. For instance, here is the first few words of ChatGPT when prompted with this question.

As of current behavior in cohttp-eio-client, yes, it does reuse already established connections when making multiple requests to the same host, provided certain conditions are met.

It’s a lie. Don’t trust them. They don’t reuse existing HTTP connection.

I was very doubtful, so I asked them how to check this. tcpdump was mentionedI later discovered eio-trace and it would have been much more straightforward to use this tool to inspect Cohttp_eio.Client’s default behavior. No LLM thought of that, sadly. . I got traces I couldn’t read at first glance, so I just copy/pasted them to the LLMs… and sure enough, they confirmed what I suspected. Cohttp_eio.Client does not share connections by default. It creates a socket for each request.

It’s actually pretty easy to convince yourself that it is the case by reading the implementation of Cohttp_eio.Client .

type connection = Eio.Flow.two_way_ty r
type t = sw:Switch.t -> Uri.t -> connection

(* simplified version of [make], omitting the support for HTTPS *)
let make () net : t = fun ~sw uri ->
  (Eio.Net.connect ~sw net (unix_address uri) :> connection)

There is nothing here dealing with persistent connections. Eio.Net.connect uses a switch for resource management, but does not perform any kind of connection caching.

That’s okay, though. Yak shaving is a real thing. I can stop working on my Ollama client library for a while, just to fix this.

The Questionable Side Quest of Implementing a Connection Pool for `cohttp-eio`

The bottom-line of this little adventure is: I should have updated my default prompt to remind the LLMs that Cohttp_eio.Body.drain in not a thing.

But let’s start from the beginning. Over the course of a few days, I have successfully implemented a wrapper on top of Cohttp_eio.Client to deal with persistent connections. It’s not rocket science, but it’s still a subtle endeavor, which necessitated a good understanding of Eio and cohttp. I cannot say LLMs were instrumental for the task. They gave me good pointers to start from, but they also misled me a bunch of times.

Sometimes, the help came in surprising ways. One anecdote in particular stuck with me. I decided I needed a get operation for Eio.Pool pools, which sadly only proposes use.

(* Provided by Eio.Pool *)
val use : 'a t -> ('a -> 'b) -> 'b

(* Not provided *)
val get : sw:Switch.t -> 'a t -> 'a

The key insight is that get allows callers to pick something from the pool, and only put it back when the switch is released.

My first implementation of get was roughly as followsI didn’t even consider asking an LLM to propose me an implementation, now that I think about it. I really am no vibe coder yet. .

open Eio.Std

let get ~sw t =
  let x, rx = Promise.create () in
  let never, _ = Promise.create () in
  Fiber.fork ~sw (fun () ->
      Eio.Pool.use t @@ fun conn ->
      Promise.resolve rx conn;
      Promise.await never);
  Promise.await x

And it didn’t work. The resulting program was hanging, because of how Fiber.fork ~sw works. Basically, the fiber created by fork becomes part of the set of fibers the switch sw waits for. Since, in my case, said fiber would never be resolved, I had created a deadlock.

I asked Gemini Pro 2.5 for help, and out of curiosity, I looked at its reasoning steps. Very early on, it mentioned Fiber.fork_daemon, but surprisingly Fiber.fork_daemon was not mentioned in the final answerOnce again, I had asked the wrong question. I asked for the Fiber equivalent of Lwt.async. I had overlooked that Lwt.async had a very particular behavior wrt. exceptions, that Gemini Pro tried very hard to replicate. I didn’t care at all about the exceptions I could raise, here! . Have I not been curious at that time, I would have missed the correct solution@alice provided me the answer a few minutes later, so I’d have been fine in the end 😅. .

I think my experience overall was made a little more frustrating than it should have been because I have never constructed a “context” that I could share between coding sessions. I haven’t enabled the memory saving setting in ChatGPT. Besides, everytime I opened Neovim, Gemini was starting from scratch. I should try to change that, to prevent the LLMs from doing the same mistakes again and again—typically, the Cohttp_eio.Body.drain function they kept bringing up.

Tip

I need to investigate how I can specialize my default prompt for each software project I am working on. I imagine I can rely on an environment variable and direnv .

Finally, it’s when I worked on this library that I came up with a nice prompt for Gemini to write my git commit messages for me.

@editor #buffer Add a git commit title and message. Structure the description in three sections (What, Why, How). Wrap the sections at 72 columns. Don’t forget the git title, and always insert a new line between the title and the description.

This prompt gives pretty cool result. It is still necessary to review it, because in a few instances I caught false statement in the proposal. But overall, it gives really meaningful output. Almost all commits of the library have been written with this prompt.

Tip

If anything, I don’t think I will never open a Merge Request with an empty description ever again.

And that, kids, is how I released cohttp-connpool-eio.0.1 .

Wrapping-up a Minimal Ollama Chat

Integrating cohttp-connpool-eio in my ocaml-ollama project led me to find a bug in the former. More specifically, the Cohttp_connpool_eio.warm function that can be used to pre-populate a new pool was doing so by performing a specified HEAD request to the host as many time as the pool sizeIn a later iteration of the library, warm only establishes connections, and does not perform any unnecessary HTTP requests. .

It worked well against both https://www.google.com and https://soap.coffee/~lthms, but when I tried with the Ollama server, it decided to hang. Why?

Well, I tried asking my new friends the LLMs, but didn’t get any answer I felt confident with. At this point, my trust in their EIO expertise was rather low, and I was more skimming through their answer to find a lead I would follow myself than anything else. In the end, I completely dropped the LLMs here, and went back to what I usually do: experimenting, and reading code.

I reproduced the issue with curl: curl -X HEAD hangs as well with Ollama, while curl --head does not. The former tries to read the response body, based on the response headers (e.g., content-length). The latter doesn’t, because it knows HEAD always omits the body. I am not sure why the hanging behavior does not show for curl -X HEAD https://www.google.com, though.

But anyway, once the bug was fixed, I could return to playing with Ollama.

I then decided to implement a helper to call POST /api/generate . It is the simplest way with Ollama to generate an LLM’s answer from a prompt. Interestingly enough, it is a “streamed” RPC using the application/x-ndjson content type. Instead of computing the answer before sending it to the client, the server instead sends JSON-encoded chunks (transfer-encoding: chunked ).

I tried to implement that with cohttp-eio, and it failed miserably with obscure parsing error messages.

After a bit of debugging, it became clear that Eio.Buf_read.parse was not behaving as I thought it was, which made me feel paranoid about how cohttp-connpool-eio handles connection releases. In the end, I had to unpack how the Cohttp_eio.Body.t work under the hood wrt. End_of_file to move on. Once again, my LLM friends weren’t particularly helpful: they were hallucinating Buf_read functions, and never considered to mention that parse only works for complete response.

Tip

My personal conclusion is that ChatGPT and Gemini quickly show their limits for non-trivial programming task involving Eio and its ecosystem. I am really curious to understand why. Do they keep hallucinating functions because Eio is a really generic name, and maybe they are mixing context from the Python library with the OCaml one? Or is it because the API of Eio has changed a lot over the years?

I am also wondering how, as a the author of a library, I can fix a similar situation. Assuming ChatGPT starts assuming false statements about cohttp-connpool-eio for instance, how do I address this? I suspect being “LLMs-friendly” will be increasingly important for a software library’s success.

In the end, ChatGPT and Gemini were just another source of inputs, not the main driver of my development process.

Putting Everything Together

Turns out, you really need just one RPC to generate a summary for a text input, so it wasn’t long before I could chain everything. I pulled mistral:7b-instruct-v0.2-q4_K_M (over a suggestion by ChatGPT, if I remember correctly), and got a summary from the video I had downloaded.

Just kidding. Out of nowhere, I decided to pursue yet another side quest, and gave a try to the fancy dune pkg lock command. Then I was able to generate my summary, using the following prompt.

Generate a summary of the raw transcript of a video provided after
this paragraph. The transcript may be in a language that is not
English, but the summary should always be in English. You should
adopt a neutral point of view (i.e., even if the transcript speaks
in the first person, you should always use the third person). Each
line is an utterance. Keep the summary short and engaging, your
goal is to provide a good overview of what was said.

----

{Vosk output}

And with this, it was time to wrap-up. And what better way to do so than to write this little journal entry? So I did, and when I was half way through my first draft, I fired a new chat buffer to ask for advice from my new friend Gemini Flash.

#buffer Here is a very preliminary, incomplete draft of a blogpost. Can you try to anticipate if it will find an audience?

It wasn’t long before Gemini turned me down.

My apologies, but I cannot anticipate whether this blog post will find an audience. My capabilities are focused on programming-related tasks like code explanation, review, generation, and tool execution within the Neovim environment.

Let me know if you have any questions about the code itself, or need assistance with Neovim.

🥲Fortunately, ChatGPT was less opinionated. .

Final Words

Although I had already used ChatGPT and other models in the past, this was the first time I tried to make them a central part of my workflow. I learned a lot during this experiment, and I now have an integrated setup I enjoy using.

I need to keep digging. Try more models (there are a lot of those now). And get better at writing good prompts which do not lead the LLMs astray. They are here to stay, after all. I better learn how to take the most from them.

What Happened in 2024?

January 1, 2025

What Happened in 2024?

Free and Open Source Software

I’ve been a “prolific contributor” at $WORK, but less so with my personal projects.

Spatial Shell remains my most “popular” projectWe should reach 100 stars on GitHub in 2025 😅. , but except for a very minor 7th release in January, I have not touched it. It’s basically a done project, and I very much enjoy using it on a daily basis. My main regret is that, contrary to what is stated in its README, Spatial Shell does not work at all with i3.
ezjsonm-encoding was initially written for Spatial Shell, but I turned it into its own OCaml package in 2024. It is a JSON-only encoding library heavily borrowing on Data-encoding API , but with a more flexible default behavior for object parsing. I enjoyed writing the documentation, inspired by a tweet from Dmitrii Kovanikov Yes, cool kids moved to Bluesky in 2024, and Dmitrii is definitely a cool kid. . That being said, I have never bothered to benchmark this package properly, so if performances are important, it may not be a good fit for you.
jsonrpc2 is, as of January 1st, 2025, an experiment in providing a general-purpose framework for servers and clients communicating with the JSON RPC 2.0 protocol. I quite like the API I’m proposing there, and maybe I’ll try to polish and publish it in 2025.
bepo-tsrn.nvim is another thing I have done for myself, but published as if it was a public good. Now, instead of having to copy/paste the same Neovim configuration file on every computer I use, I can just type yay -S neovim-bepo-tsrn-git and be done with it.
celtchar has seen its first commits since 2021, which is not nothing. As a reminder, celtchar is a little tool I have written to generate ebooks and static websites for the stories I write; and as I was doing the 2024 edition of NaNoWriMo , I found myself in need to add a missing feature (supporting books split in parts, not only chapters).

Overall, I’ve been defaulting to OCaml for the past two years or so, and I am starting to think it is time to widen my perspective again. I will probably start with relearning GoI don’t know why, but I have been mildly obsessed with this language lately. .

Blog posts

2024 was not a very productive year when it comes to this website. I have published 5 articles, which is half the number of publications of 2023. As a logical consequence, not a lot of folks have visited my website this year. Funnily enough, the most read article (by far) in 2024 was published in 2023To be fair, it was published on December 30, 2023. .

Yes, 2024 was a quiet year for this website

That being said, I am quite happy with the content published in 2024.

Using git maintenance with Encrypted SSH Keys is the first article I published in 2024. It is a direct consequence of my trip to Brussels in February to attend to FOSDEM . If you haven’t already, you should watch Scott Chacon’s talk about Git less known commands; it is the only reason why I learned about git maintenance.
Installing a LUKS-Encrypted Arch Linux on a Vultr VPS is mostly a gift I have made to Future Me. It is a very specific how-to that I can use to quickly set up a new server with disk encryption. Funny story, I was planning to publish a follow-up about how I use systemd-nspawn to run my web services in containers , but no matter how many times I tried, I’ve never quite found a good way to tell this story. As I plan to educate myself on Kubernetes in 2025, it is not clear I will ever publish it now.
Introducing bepo-tsrn.nvim is probably the less useful article I have published in 2024, considering I expect the userbase to bepo-tsrn.nvim to stick to 1 until the very endBut who knows? Maybe one of you will prove me wrong! .
On Vestigial Structures hardly qualifies as a blog post, and is mostly a joke. It is also the only content on my website that was mostly generated by ChatGPT, and it is flagged as such. I don’t like using AI to write, but I do appreciate having a reviewer always at hand.
Serving This Article from RAM for Fun and No Real Benefit was very fun to write. This little experiment was stuck in my head for basically two years, and it turned out basically exactly as I had pictured it. That being said, I want to learn about CDNs now.

Overall, I still enjoy having my own little corner of the Internet, but if there is one thing I’d like to improve in 2025, it is its reach. I’d like you folks to run into my website, instead of having to promote it every time I write something. 2025, the year of SEO?

`$WORK`

2024 started with my decision to go back to a Software Engineering position, after giving an honest try at being an Engineering Manager in late 2023.

I want to remember 2024 for two things.

This year, more than ever, I have tried to appreciate my work beyond my individual contributions. I am confident in my programming skills (although I have so much to learn), but being an accomplished engineer is much more than contributing code. Making sure every engineer in the team can work to the best of their current ability, fostering a work environment favoring growth and initiative, estimating as precisely as possible the amount of time needed to deliver the next important thing, collaborating efficiently with non-technical teams... I am becoming increasingly interested in these areas.

Besides, this year was all about delivering and deploying in production. It’s been a wild ride , and I loved it even if it was very demanding. After having mostly contributed to R&D projects, the focus on UX, backward compatibility, etc. was very refreshing. I learned so much through the year, and had many opportunities to make significant impacts.

In 2025, I want to keep learning about software engineering, and maybe start sharing my thoughts on the subject on my website.

Talks

I gave only one talk in 2024, at a conference called EthCC . It was actually a follow-up to the talk I gave the year before. You can watch me deliver the talk here , but if you are more into written content, I have also published a transcript on this very website. I actually loved writing it down, and plan to systematically publish similar content for every recorded talk I will give in the future.

I also had the opportunity to participate in a Twitter Space .

This year, my public speaking opportunities were all $WORK-related. I would like to change this in the future, because there are enough events out there for me to start speaking about something other than workIt’s too late to apply to FOSDEM, but maybe I can find something to say to an event later in the year! I think. .

Sport

This year was a bit of a disappointment, sport-wise. I tried several times to get back to running regularly, and failed miserably. I went to Lyon for a 10km run without proper training, and skipped the half-marathon I signed up for. I should update my Running Log nonetheless. I started swimming regularly during the Summer, only to pierce my earlobes in September 😅.

Let’s hope I do better in 2025! I am planning to register for the Triathlon de Deauville with my sister. That promises to be fun! And I want to commit to the 20km de Paris as well.

On the bright side, I have started to use my bike again. I love riding around Paris, especially at night.

Final Notes

Overall, I’ve devoted a large part of my time to $WORK in 2024. Hopefully, I will find a better balance over the course of 2025, which should give me more time to explore and experiment more things.

Anyway, happy new year everyone! And happy Dry January!

Serving This Article from RAM for Fun and No Real Benefit

December 25, 2024

Serving This Article from RAM for Fun and No Real Benefit

docker meta ocaml

In 2022, Xe Iaso published a transcript of their talk on how their website was working at the time . In a nutshell, their approach consisted of a server preprocessing the website from its source at startup, then serving its contents from memory. If you have not already, I can only encourage you to read the article or watch the talk, as the story they tell is very interesting. For me personally, it sparked a question: what if, instead of preprocessing the website at startup, one decided to embed the already preprocessed website within the program of the HTTP server tasked to serve it?

Fast-forward today, and this question has finally been answered. The webpage you are currently reading has been served to you by an ad hoc HTTP server built with Dream , whose binary is the only file I need to push to my server to deploy the latest version of my website. I have actually deployed it, and it’s been serving the contents of this website for more than a week now.

What did I learn from this fun, little experiment? Basically, that this approach changes nothing, as far as Lighthouse and my monitoring is concerned. I couldn’t find any meaningful differences between a static website served by Nginx, a piece of software with thousands and thousands of engineering work behind it, and my little toy web server pieced together in an hour or so. Still. It was fun, so why not write about it?

This article is a kind of experience report. I’ll dive into what I have done to turn my website into a single, static binary. Not only does it mean writing some OCaml, which is always fun, but it also requires understanding a little some key HTTP headers, as well as using Docker to build easily deployable binaries. All in all, I hope it will be an interesting read for the curious minds.

Embedding My Website in a Binary

Not much had changed much since I stopped using cleopatra to generate this website, and the article I published in 2023 still stands. In a nutshell, I work in the site/ directory, and soupault generates my website in the out/~lthms directory, thanks to a collection of built-in and ad hoc pluginsFor instance, Markdown footnotes are turned into side notes with a soupault plugin. . To deploy the website, I was relying on rsync to sync the contents of the out/~lthms directory with the directory statically served by a Nginx instance on my personal server.

The first step of my little toy project is to actually embed the output of soupault into an OCaml program.

That’s where ocaml-crunch comes in handy. It is a MirageOS project, whose only job is to generate an OCaml module from a file system directory. It is straightforward to use it from Dune.

; file: out/dune
(rule
 (target website_content.ml)
 (deps (source_tree ~lthms))
 (action
  (run ocaml-crunch -m plain -o %{target} -s ~lthms)))

This snippet generates the website_content.ml module, which we can then expose through a library with the library stanza.

; file: out/dune
(library
 (name website_content))

And we are basically done. Excluding an Internal module, the signature of Website_content is pretty straightforward.

val file_list : string list
val read : string -> string option
val hash : string -> string option
val size : string -> int option

Serving the content with Dream

Dream is a cool project, and provides a straightforward API that we can leverage to turn our list of in-memory files into an HTTP server.

Naive Approach

Our goal now is to create a Dream.handler for each item in file_list. Done naively (as was my first attempt), it gives you something of the form:

let make_handler ~content path =
  Dream.get path (fun req ->
    Lwt.return (Dream.response content)))

Which we can use to build the main route we will then pass to Dream.router.

let website_route =
  Dream.scope "~lthms" []
  @@ List.map
       (fun path ->
         let content = Option.get (Website_content.read path) in
         make_handler ~content path)
       Website_content.file_list

With this approach, we build our handlers once, and then the lookup is done by Dream’s router. It could be an interesting experiment to see if doing the lookup ourselves is more performant (since Dream’s router is very generic, while in our case we don’t really need to parse anything). I remember Xe routing is basically going through a linked list, which seems strange at first, but works very well in practice because they have ordered said list with the most recent articles up front, and everybody comes to their website to read the latest article anyway.

It does not take an extensive QA process to figure out that this approach is far from being enough. To name a few things:

My website assumes http://path/index.html can be accessed with http://path/ or http://path. Our little snippet does not handle this.
Browsers expect the Content-Type headers to be correctly set. To give an example, they won't load a CSS file if the Content-Type header is not set to text/css.
Browsers work best for websites that take the time to provide caching directives. Our little snippet does not care to do so.
Even if my website is rather lightweight20MBytes at the time of writing the first version of this article. , compressing the response of our HTTP server for clients that support it is always a good idea.

This is a gentle reminder of all the things Nginx can do for you with very little configuration.

Handling `index.html` Synonyms

This one is rather simple. For files named index.html, we need 3 handers, not just one. We can achieve this with an additional helper make_handler_remove_suffix.

let make_handler_remove_suffix ~content path suffix
    =
  if String.ends_with ~suffix path then
    let alt_path =
      String.sub path 0 (String.length path - String.length suffix)
    in
    [ make_handler ~content alt_path ]
  else []

Updating the website_route definition to use make_handler_remove_suffix is quite easy as well.

 let website_route =
   Dream.scope "~lthms" []
-  @@ List.map
+  @@ List.concat_map
        (fun path ->
          let content = Option.get (Website_content.read path) in
-         make_handler ~content path)
+         if path = "index.html" then
+           (* Special case to deal with "index.html" which needs to be
+              recognized by the route "/" *)
+           [
+             make_handler ~content "/";
+             make_handler ~content "";
+             make_handler ~content "index.html";
+           ]
+         else
+           make_handler_remove_suffix ~content path
+             "/index.html"
+           @ make_handler_remove_suffix ~content
+               path "index.html"
+           @ [ make_handler ~content path ])
        Website_content.file_list

With that, https://soap.coffee/~lthms/posts/index.html returns the same pages as https://soap.coffee/~lthms/posts or https://soap.coffee/~lthms/posts. Check.

Supporting `Content-Type`

Content-Type is an HTTP header which is used by the receiver of the HTTP message (whether it is a request or a response) to interpret its content.

For instance, when building a RPC API, Content-Type is used by the server to know how to parse the request body (Content-Type: application/json or Content-Type: application/octet-stream being two popular choices, for JSON or binary encoding, respectively).

In our case, the Content-Type header is used by the HTTP server to communicate the nature of the content to browsers. For my website, I can just use the file extensions to infer the correct header to set. First, we list the extensions that are actually used.

let content_types =
  [
    (".html", "text/html");
    (".css", "text/css");
    (".xml", "text/xml");
    (".png", "image/png");
    (".svg", "image/svg+xml");
    (".gz", "application/gzip");
    (".pub", "text/plain");
  ]

A header in Dream is encoded as a string * string value, with the first string being the header name and the second being the header value.

let content_type_header path =
  List.filter_map
    (fun (ext, content_type) ->
      if String.ends_with ~suffix:ext path then
        Some ("Content-Type", content_type)
      else None)
    content_types
  |> assert_f
       ~error_msg:Format.(sprintf "Unsupported file type %s" path)
       (( <> ) [])

with assert_f being defined as follows.

let assert_f ~error_msg f v =
  if f v then v else failwith error_msg

assert_f is used to enforce that I don’t deploy a website which contains route lacking a Content-Type header. For instance, if I remove the "html" entry of the content_type list, I get this error when I try to execute the server.

Fatal error: exception Failure("Unsupported file type index.html")

This is because the headers are only computed once, when each route are defined. This is a key principle of this project: compute once, serve many timeI would love to get a compilation error instead (considering there are no runtime values involved), but have not looked into this just yet. .

 let website_route =
   Dream.scope "~lthms" []
   @@ List.concat_map
        (fun path ->
          let content = Option.get (Website_content.read path) in
+         let headers = content_type_header path in
          if path = "index.html" then
            (* Special case to deal with "index.html" which needs to be
               recognized by the route "/" *)
            [
-             make_handler ~content "/";
-             make_handler ~content "";
-             make_handler ~content "index.html";
+             make_handler ~headers ~content "/";
+             make_handler ~headers ~content "";
+             make_handler ~headers ~content "index.html";
            ]
          else
-           make_handler_remove_suffix ~content path
+           make_handler_remove_suffix ~headers ~content path
              "/index.html"
-           @ make_handler_remove_suffix ~content
+           @ make_handler_remove_suffix ~headers ~content
                path "index.html"
-           @ [ make_handler ~content path ])
+           @ [ make_handler ~headers ~content path ])
        Website_content.file_list

(The changes in make_handler and make_handler_remove_prefix are left as an exercise to enthusiast readers)

Compressing if Requested

Nowadays, computations are cheap, while downloading data costs time (and sometimes money). As a consequence, it is often a good idea for a server to compress a large HTTP response, and browsers do ask them to do so, by setting the Accept-Encoding header of their requests.

The value of the Accept-Encoding header is a comma-separated list of supported compression algorithms, optionally ordered with a priority value q.

For instance, Accept-Encoding: gzip;q=0.5, deflate;q=0.3, identity tells you that the browser supports three encoding methods: gzip, deflate and identity (no compression), and the browser prefers gzip over deflate. Besides, the request can provide several Accept-Encoding headers instead of just one, so we can have

Accept-Encoding: gzip;q=0.5
Accept-Encoding: deflate;q=0.3
Accept-Encoding: identity

The String module provides everything we need to check if a browser supports gzip as an encoding methodSpoiler: they do. I was even wondering at some point if I could just always return GZIP-compressed values, ignoring the Accept-Encoding header altogether. If you do that, though, curl becomes annoying to use (it does not uncompress the response automatically, and instead complains about being about to write binary to the standard output). .

(* For [method(; q=val)?], returns [method], except if
   [q=0]. *)
let to_directive str =
  match String.split_on_char ';' str |> List.map String.trim with
  | [ x ] -> Some x
  | [ x; y ] -> (
      match String.split_on_char '=' y |> List.map String.trim with
      | [ "q"; "0" ] -> None
      | [ "q"; _ ] -> Some x
      | _ -> None)
  | _ -> None

(* [contains ~value:v header] returns [true] if [v] is a
   supported method listed in [header]. *)
let contains ~value header =
  String.split_on_char ',' header
  |> List.to_seq |> Seq.map String.trim
  |> Seq.filter_map to_directive
  |> Seq.exists (( = ) value)

We use contains to tell us if we can return a compressed response, which leaves us with one final question: how to compress said response?

The OCaml ecosystem seems to have picked camlzip library when GZIP is involvedYou know it is a legitimate OCaml library when one of the top-level modules is not documented at all . . What is surprising with this library is that it does not support in-memory compression: the functions expect channels, not bytes. That is quite annoying, because we are specifically doing this not to use files.

The Internet is helpful here, and quickly suggests using pipes. It works when you remember –or figure out– that pipes are a blocking mechanism: one does not just write a buffer of arbitrary size in a pipe, because after something like 4KBytes, writing becomes blocking until a read happens to free some space. That’s not a big problem: we can read and write concurrently to the pipe using threads, and OCaml 5 makes it quite easy to do so with the Domain module.

let gzip content =
  let inc, ouc = Unix.pipe () in
  let ouc = Gzip.open_out_chan ~level:6 Unix.(out_channel_of_descr ouc) in
  let _writer =
    Domain.spawn (fun () ->
        Gzip.output_substring ouc content 0 String.(length content);
        Gzip.close_out ouc)
  in
  let res = In_channel.input_all Unix.(in_channel_of_descr inc) in
  Unix.close inc;
  res

Caution

As rightfully pointed out by Daniel Bünzli , the gzip function presented in this article is full of shortcomings. To quote the message, the function can leak fds in case of errors and domains are not meant to be used that way (it’s rather spawn one long running domain per CPU you have). It’s not necessarily more complicated to correct it to use Fun.protect invocations to make sure all your fds get closed even if the function blows up and use Thread.create so that the netizens cut and paste correct code.

In my opinion, this implementation is “good enough” for my use case, which is compressing arbitrary strings before the HTTP server is even started. If it were to be called in the handlers themselves, then definitely, it would not be suitable.

Keep that in mind if you want to borrow this code.

We know how to decide whether to compress or not, and how to compress. The next step is to modify make_handler accordingly.

-let make_handler ~headers ~content path =
+let make_handler ~headers ~gzip_content ~content path =
+  let gzip_headers = ("Content-Encoding", "gzip") :: headers in
   Dream.get path (fun req ->
-    Lwt.return (Dream.response content)))
+      match Dream.headers req "Accept-Encoding" with
+      | accepted_encodings
+        when List.exists (contains ~value:"gzip") accepted_encodings ->
+          Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
+      | _ -> Lwt.return @@ Dream.response ~headers content)

gzip_content is computed only once (using our gzip function), and passed to make_handler. This way, the only computation the handler needs to do is to “parse” Accept-Encoding.

Caching

Compressing a page to reduce the number of bytes a browser needs to download is fine. Letting the browser know it does not need to download anything because it's previous version is still accurate is better.

This is achieved through two complementary mechanisms: entity tags (ETag)My first encounter with entity tags was around the time GDPR was a hot topic, because you can use them as a cheap replacement for cookies to track your users . I remained at the surface level at the time, it was fun learning more about them through this little project. , and cache policies (Cache-Control).

Entity tags are used to identify a resource, and are expected to change every time the resource is updated. The general workflow goes like this: the first time a browser requests https://soap.coffee/~lthms/index.html, it caches the result along with the value of the ETag header. The next time it needs the page, it adds the header If-None-Match, with the ETag as its value. For such requests, the server is expected to return an empty response with HTTP code 304 (Not Modified).

I decided to use the sha256 hash algorithm to compute the entity tag of each resource of my website. The sha OCaml library looked like a good enough candidate.

(* in `website_route` *)
let etag = Sha256.(string content |> to_hex) in

Interestingly, one question I had to answer was whether the entity tag of a page needed to be different whether it was compressed or not. Internet almost unanimously answered yes. So be it. We just need to keep in mind ETag values are expected to be surrounded by quotes, and we are good to go. It is just a matter of suffixing the ETag with +gzip in the compressed case.

-let make_handler ~headers ~gzip_content ~content path =
-  let gzip_headers = ("Content-Encoding", "gzip") :: headers in
+let make_handler ~headers ~etag ~gzip_content ~content path =
+  let etag_gzip = Format.sprintf "\"%s+gzip\"" etag in
+  let etag = Format.sprintf "\"%s\"" etag in
+  let gzip_headers =
+    ("Content-Encoding", "gzip") :: ("ETag", etag_gzip) :: headers in
+  let identity_headers = ("ETag", etag) :: headers in
   Dream.get path (fun req ->
-      match Dream.headers req "Accept-Encoding" with
-      | accepted_encodings
-        when List.exists (contains ~value:"gzip") accepted_encodings ->
-          Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
-      | _ -> Lwt.return @@ Dream.response ~headers content)
+      match Dream.headers req "If-None-Match" with
+      | [ previous_etag ] when previous_etag = etag || previous_etag = etag_gzip
+        ->
+          Lwt.return
+          @@ Dream.response
+               ~headers:(("ETag", previous_etag) :: headers)
+               ~code:304 ""
+      | _ -> (
+          match Dream.headers req "Accept-Encoding" with
+          | accepted_encodings
+            when List.exists (contains ~value:"gzip") accepted_encodings ->
+              Lwt.return @@ Dream.response ~headers:gzip_headers gzip_content
+          | _ -> Lwt.return @@ Dream.response ~headers:identity_headers content))

Entity tags are useful, but you still need the browser to make an HTTP request every single time you visit the website. By setting a cache policy, we can remove even remove the need for this request most of the time. The Cache-Control header is used to set a number of parameters, including the max-age value (in seconds).

In my Nginx configuration, I had set max-age to a year for images. I did the same thing here. Besides, I decided to set max-age for other resources to 5 minutes. This seems like a good compromise: since my website does not change very often, it is very unlikely that you happen to visit it when I publish new content. Setting a 5-minute cache policy should let my readers download each resource only once, yet get the freshest version at their next visitDealing with the Cache-Control header is basically the same exercise as setting the correct Content-Type header, and this article is already long enough, which is why there is no diff or snippet in this section. .

And with this, we are done. We get a standalone library to server our website in a browser-friendly manner, which I can theoretically use to replace my current Nginx-powered setup. Although… is it that simple?

Building and Deploying the Website

Files are quite easy to share and deploy. As I mentioned earlier in this article, you just need to rsync them and be done with it. Binaries, on the other hand… One cannot just assume a binary build on a machine X will work on another machine Y. Some additional works need to be done.

The most straightforward solution I know is to rely on static binaries. I have already written about how to generate static binaries for OCaml projects, so I had a pretty strong head start, but one drawback of the approach I’ve described there (and that I have been using for Spatial Shell’s releases Not that there were many of them lately. ) is that it is rather slow (I have a script creating a new local switch each time) and requires static libraries to be installed (which Arch Linux does not provide).

And so, I figured, why not build the static binary in Docker? This allows me to use Alpine to get a static version of my system dependencies, and can be quite fast thanks to Docker build cache . The Dockerfile is quite simple: one stage for building the system and OCaml dependencies, and one stage for building the static binary.

FROM alpine:3.21 AS build_environment

# Use alpine /bin/ash and set shell options
# See https://docs.docker.com/build/building/best-practices/#using-pipes
SHELL ["/bin/ash", "-euo", "pipefail", "-c"]

USER root
WORKDIR /root

RUN apk add autoconf automake bash build-base ca-certificates opam gcc \ 
  git rsync gmp-dev libev-dev openssl-libs-static pkgconf zlib-static \
  openssl-dev zlib-dev
RUN opam init --bare --yes --disable-sandboxing
COPY makefile dune-project .
RUN make _opam/.init OCAML="ocaml-option-static,ocaml-option-no-compression,ocaml.5.2.1"
RUN eval $(opam env) && make server-deps

FROM build_environment AS builder

COPY server ./server
COPY out ./out
COPY dune .
RUN eval $(opam env) && dune build server/main.exe --profile=static

FROM alpine:3.21 AS soap.coffee

COPY --from=builder /root/_build/default/server/main.exe /bin/soap.coffee

Then, building my static binary becomes as simple as:

docker build . -f ./build.Dockerfile \
  --target soap.coffee \
  -t soap.coffee:latest
docker create --name soap-coffee-build soap.coffee:latest
docker cp soap-coffee-build:/bin/soap.coffee .
docker rm -f soap-coffee-build

docker cp does not work on an image, but on a container, so we need to create one which can be destroyed shortly after.

This little binary weights 38MBytes, which seems relatively reasonable to me, considering my website weights 20MBytes. I guess it could be easy to reduce this size by embedding the compressed version of my articles and images, instead of the uncompressed one. But really, for my website, I’m not really interested in investing the extra effort.

Conclusion

I would not recommend anyone to use this in production for anything remotely important, but from my perspective, it was both fun and insightful. I was able to refresh my memories about HTTP “internal,” among other things. Again, as far as I can tell, deploying my website this way did not bring me any benefit, performance-wise; even worse, I am pretty sure the Dream server will not behave as well as Nginx when it comes to handling the load (since it is limited to one core, instead of several with Nginx).

That being said, I am way too invested now… which is why, yes, you are reading a blog post served to you directly from memory 🎉.

On Vestigial Structures

August 16, 2024

On Vestigial Structures

satire ocaml

Do you know what vestigial structures are?This write-up is a slightly modified version of a commit description I have written for $WORK with some help from ChatGPT .

In a nutshell, they are remnants of structures that were functional in an ancestral species but have lost much or all of their original function in the descendant species.

For instance, some snakes, like pythons and boas, have tiny remnants of hind leg bones, which are vestigial structures from when their ancestors had legs. Birds like ostriches and emus have wings that are too small for flight. These wings are vestigial structures as well, remnants of their flying ancestors. Even humans have some, like those small muscles around their ears. These muscles were once used by our ancestors to move their ears, similar to how many animals can today.

Sometimes, vestigial structures are not just useless, they can be harmful. Take our wisdom teeth. They were handy to our ancestors, who had larger jaws and a diet that required more chewing. Sadly, they have become problematic for many modern humans.

Anyway, ocaml-migrate-parsetree is a deprecated OCaml library in $WORK’s dependency tree that will never be compatible with OCaml 5, but it turns out we do not need it anymore .

Introducing bepo-tsrn.nvim

May 23, 2024

Introducing `bepo-tsrn.nvim`

neovim releases

It is not much, but I have recently published on GitHub what can be considered my very first Neovim “plugin.”

bepo-tsrn.nvim is a zero-configuration, global plugin for Neovim which remaps default Neovim bindings for the Bépo keyboard layout. It started as a fork of bepo.nvim with two significant changes:

hjkl are remapped to tsrn instead of ctsr. hjkl as motion keys always had one great drawback in my opinion: they required me to shift my left hand whenever I wanted to use them. Well, no more!
bepo-tsrn.nvim is a global plugin, unconditionally loaded at startup without the need to explicitely load it. As a consequence, it is a very convenient way to get Bépo enabled in Neovim system-wide. Bonus point if you are using Archlinux, since I have added it to the Archlinux User Repository (AUR).

bepo-tsrn.nvim is released under the terms of the Apache-2.0 license (inherited from bepo.nvim ). Granted, this plugin targets a very small niche of users, but maybe you will find it as useful as I do!

Installing a LUKS-Encrypted Arch Linux on a Vultr VPS

February 25, 2024

Installing a LUKS-Encrypted Arch Linux on a Vultr VPS

self-hosting vultr

I’ve been a happy customer of Vultr for three years now. For one, this little corner of the Internet is hosted on one of their VPS, along with other services I have self-hosted. Recently, I have decided to migrate to a new VPS with more disk space and more powerful vCPUs.

In this article, I describe how I have set up the host (an Arch Linux system on a LUKS-encrypted partition) of jasminejasmine being my shiny new server—I have taken the habit to name my servers and personal computers after Disney princesses. . I had done a similar thing for mulan, jasmine’s predecessor. Unfortunately, I didn’t take the time to document mulan installation steps three years ago. Needless to say, I had plenty of opportunities to regret this oversight while I painfully redicovered how to make this work.

Hopefully, most of what is in this write-up will stand the test of time and remain true three years from now, when I migrate again 😅.

Booting on the Installation ISO

It is no surprise that Vultr does not propose an out-of-the-box LUKS-encrypted Arch Linux installation in their OSes library. As a consequence, I had to boot my brand new VPS on the Arch Linux ISO .

In the Vultr dashboard, I visited the “ISOs” page (in the “Orchestrations” menu) to add the latest Arch Linux installation ISO.
Then, I went to the management panel of jasmine, more precisely in the “Settings” tab. I visited the “Custom ISO” page, and attached the Arch Linux ISO. This prompted a reboot.

jasmine happily booted the image, but the resulting system was lacking a SSH server to connect to. Instead, I used the “View Console” feature to get a shell access to my VPSIt is noteworthy that I had to disable one of my Firefox extensions: Vimium. Vimium provides me vim-like keystrokes in the browser, and the keys I am using for motion (t, s, r, and n) were not forwarded to the console. .

The Arch Linux installation ISO’s default keymap is qwerty, a keyboard layout I am helpless with. The first command I always use in this situation is loadkeys, which allows me to switch to the fr-bepo layout.

Note

In this article, I am using ; as the terminal prompt for ease of copy/pasting .

; loadkeys fr-bepo

Remote Access to `jasmine`

The next step was to start a SSH server I could connect to using my terminal emulator. A virtual console is nice enough, but when your muscle memory commands you to hit Ctrl-W to delete the word before your cursor as soon as you are connected to a shell, you don’t want to stay in a browser tab for obvious reasons.

I set up a password for the root user using passwd.
I modified the /etc/ssh/sshd_config file to let me connect with the root user using a password (PermitRootLogin yes).
I started the SSH server using systemctl start sshd.

From there, I could connect jasmine from my terminal emulator.

# after modifying /etc/hosts to add `jasmine` IP there
; ssh root@jasmine

Disk Partitions

Now that I was enjoying the comfort of my terminal emulator of choice (kitty ), the serious business could start. Moving forward, my next task was to set up the disk layout and the filesystems.

Warning

Did you know that Vultr VPS are using legacy boot and not UEFI? I surely did, three years ago, but sadly I had forgotten. So I did a first installation setup with a EFI partition layout (GPT, /boot formatted with VFAT, etc.), only to discover that my shiny system could not boot.

To create an MBR partition system, fdisk is a nice enough tool. The disk is exposed to the VPS under the name /dev/vda.

; fdisk /dev/vda

Firstly, I created a new MBR.

Command (m for help): o

Then, I created the /boot partition. To be sure I would not run out of space, I allocated 3GBytes for it.

Command (m for help): n
Partition number (1-128, default 1): 
First sector (34-268435422, default = 2048) or {+-}size{KMGTP}: 
Last sector (2048-268435422, default = 268433407) or {+-}size{KMGTP}: +3G

The root partition was to take the remaining space.

Command (m for help): n
Partition number (2-128, default 2): 
First sector (34-268435422, default = 6293504) or {+-}size{KMGTP}: 
Last sector (6293504-268435422, default = 268433407) or {+-}size{KMGTP}:

I reviewed the resulting partition layout.


Command (m for help): p

Disk /dev/vda: 128 GiB, 137438953472 bytes, 268435456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdbc3620d

Device     Boot   Start       End   Sectors  Size Id Type
/dev/vda1          2048   6293503   6291456    3G 83 Linux
/dev/vda2       6293504 268435455 262141952  125G 83 Linux

LGTM!

Command (? for help): w

Filesystems

`/boot` Partition

No need to be particularly fancy here. I formatted /boot with the ext3 filesystem. As far as I know, this is a reasonable default choice for this partition.

; mkfs.ext3 /dev/vda1

LUKS-Encrypted Root Partition

I mostly followed the Arch Linux Wiki for this. Generally speaking, the Arch Linux Wiki is a wonderful source of information for anything related to Linux.

I formatted /dev/vda2 with cryptsetup to initialize the encryption scheme.

; cryptsetup -y -v luksFormat /dev/vda2

Important

Be sure to save your LUKS passphrase somewhere, otherwise you are likely to run in trouble at some point. I’ve personally added it to my Bitwarden.

As-is, the partition cannot be used; it needs to be “open” with cryptsetup.

; cryptsetup open /dev/vda2 root

root here is the name given to the usable partition. More precisely, after this command the partition can be manipulated through the /dev/mapper/root logical volume.

Before, I had always used ext4 for my root partition, but for a while I’ve had an interest for alternative filesystems. And so, I decided out of the blue to format /dev/mapper/root with btrfsTime will tell if this was a mistake… The Internet seems to suggest it might be one 😅. . I like the idea of a filesystem compressing its contents by default.

; mkfs.btrfs -d single -m single /dev/mapper/root
; mount -t btrfs -o defaults,noatime,compress=zstd /dev/mapper/root /mnt

Finally, I could mount the two formatted partitions in /mnt.

; mkdir /mnt/boot
; mount /dev/vda1 /mnt/boot

Installing Arch Linux

I have installed the bare minimum with pacstrap. I personally prefer entering in the chroot as soon as possible.

; pacstrap -K /mnt base linux
; genfstab -U /mnt >> /mnt/etc/fstab
; arch-chroot /mnt

In the rest of the article, I focus on the details that are directly related to our main subject. That is, being able to boot a LUKS-encrypted Arch Linux VPS. To that end, I have chosen to reproduce the exact setup I did three years ago for mulan. It consists in configuring the initramfs to spawn an SSH server (tinyssh ) which let me enter the LUKS passphrase of /dev/vda2 remotely.

`sshd`

Before configuring the boot process, it had to take a small detour and configure the SSH server (sshd from OpenSSH) jasmine will start once it has booted. The main reason is quite simple: I wanted tinyssh and sshd to use the same host key.

tinyssh is limited in terms of the key types it supports, but I personally always default to Ed25519 so I’m fine with that.

ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519

I then modified /etc/ssh/sshd_config to ensure sshd would default to it.

# in /etc/ssh/sshd_config, uncomment
HostKey /etc/ssh/ssh_host_ed25519_key

I used this opportunity to also (temporarily) allow login as root using a password. Eventually, I will definitely remove this line from sshd config, but while I’m configuring everything, it’s too convenient to bother with another approach.

# in /etc/ssh/sshd_config, add
PermitRootLogin yes

This line is only useful if root has a password, so I initialized one.

; passwd

I also enabled the sshd service, to let systemd start sshd automatically at boot time.

; systemctl enable sshd

TinySSH

At this point, I knew I had only dealt with the easy bits. The hard part is to be able the damn thing.

I have always used Busybox-based initial ramdisk. While rereading the Arch Linux Wiki, I discovered it was also possible to use systemd, which… well, nobody is surprised by this, I guess. I am a happy adopter of systemd multiple features in general, but for the particular setup I am targeting, using systemd means installing an AUR package . So I decided to stick with Busybox.

I gathered and installed the packages I needed for this to work.

; pacman -S tinyssh mkinitcpio-tinyssh mkinitcpio-netconf mkinitcpio-utils python3

And I modified /etc/mkinitcpio.conf to modify the HOOKS array.

-HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block filesystems fsck)
+HOOKS=(base udev autodetect modconf block netconf tinyssh encryptssh filesystems keyboard keymap fsck)

I tried to keep it minimal (hence the removal of kms for instance, which does not seem very useful for a remote server considering its about setting display resolution), and I borrowed from the /etc/mkinitcpio.conf file of mulan for netconf, tinyssh and encryptssh.

Turns out mkinitcpio-tinyssh is affected by a long standing bug , but fortunately the patch to /usr/lib/initcpio/install/tinyssh worked like a charm.

   local return_code=1
 
   if [ ! -d $destdir -a -x /usr/bin/tinyssh-convert ]; then
-      mkdir $destdir
-  fi
-  
-  if [ -s "$osshed25519" -a ! -s $destdir/.ed25519.sk -a ! -s $destdir/ed25519.pk -a -x /usr/bin/tinyssh-convert ]; then
-      tinyssh-convert -f $osshed25519 -d $destdir
+      tinyssh-convert $destdir < $osshed25519
       if [ $? -eq 0 ]; then
           return_code=0
       fi

On my local machine, I generated a brand new SSH key using the Ed25519 key type.

; ssh-keygen -t ed25519 -f jasmine

I copy/pasteed the content of jasmine.pub in /etc/tinyssh/root_key on jasmine. It was also a good idea to delete the etc/tinyssh/sshkeydir directory, before running the command to generate the initial ramkdisk.

; mkinitcpio -p linux

Grub

This is your periodic reminder that Vultr does not support UEFI and still requires to the legacy MBR to boot. This means systemd-boot is out of the picture. I always feel overwhelmed when i have to configure Grub. However, three years ago I managed to make it work, so I decided to stick to this again.

; pacman -S grub
; grub-install --target=i386-pc /dev/vda
; grub-mkconfig -o /boot/grub/grub.cfg

/boot/grub/grub.cfg contains a lot of things. I looked for the line

### BEGIN /etc/grub.d/10_linux ###

In the first menuentry after it, I modified one lineMy initial thought was to install the intel-ucode package, and to configure Grub to load Intel’s microcode upgrades at boot time. However, since jasmine is a VPS, it does not directly access the physical CPU it is running onto. As a consequence, this step is not necessary. .

First, the linux line.

        linux   /vmlinuz-linux root=/dev/mapper/root rw cryptdevice=UUID=7e58f868-529
a-4d70-bdfd-31317017c30b:root ip=<server ip>::<server gateway>:<server netmask>::eth0:none logle
vel=3 quiet

The one is the most tricky.

root=/ev/mapper/root rw is the most straightforward. It means the root filesystem in located in /dev/mapper/root, that is a logical volume.
cryptdevice=UUID=7e58f868-529a-4d70-bdfd-31317017c30b:root tells the kernel that the root logical volume is inside the encrypted (cryptdevice) /dev/vda2 partition. To get the UUID of /dev/vda2, I used blkid.

; blkid | grep UUID
/dev/mapper/root: UUID="0a44e4c7-10b7-4e85-8228-7663cf16a631" UUID_SUB="0c9edc1c-31c0-417d-93e2-6f035c68b7cb" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/vda2: UUID="e9695de0-403c-4a6c-b8a2-9891dca2a60a" TYPE="crypto_LUKS" PARTUUID="dbc3620d-02"
/dev/vda1: UUID="fa7a24d2-fc76-4af2-b26c-58548650253f" BLOCK_SIZE="4096" TYPE="ext3" PARTUUID="dbc3620d-01"

ip=<server ip>::<server gateway>:<server netmask>::eth0:none tells the kernel enough information for tinyssh to be accessible from the Internet. These three IPS can be found in the “Settings” tab of jasmine management panel in the Vultr dashboard.

Networking

There was only one step remaining before rebooting jasmine without the installation ISO attached: configuring the system to ensure remote access. In practice, it means enabling DHCP for the Ethernet interface (named ens3 in jasmine’s case).

I did that by defining a .network unit in /etc/systemd/networkd to enable DHCP so that jasmine can get its ip from Vultr DHCP server.

# /etc/systemd/network/50-ens3.network
[Match]
Name=ens3

[Network]
DHCP=yes

As a last touch, I enabled both systemd-networkd and systemd-resolved.

; systemctl enable systemd-networkd
; systemctl enable systemd-resolved
; ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

And I was good to go!

I exited the chroot, came back to Vultr dashboard to detach the custom ISO from jasmine, which prompted a reboot of the VPS. A look a the virtual console was enough to confirm tinyssh had started and was waiting for incoming connection. I edited my ~/.ssh/known_hosts to remove jasmine now out-dated line, and tried to connect to jasmine. tinyssh asked me kindly the passphrase for /dev/vda1, before closing the connection.

A new attempt at connection to jasmin using SSH led me to the expected system. Hourray!

Conclusion

Needless to say, while this article is meant to propose a linear story to its readers, my personal journey was everything but. I had to reboot many times into the installation ISO, to mount my root partition and tweak the configuration until it finally worked.

Hopefully, this article lists the various pitfalls I ran into, and how I fixed them. It’ll be helpful for Future Me, and maybe to you before that.

Using git maintenance with Encrypted SSH Keys

February 18, 2024

Using `git maintenance` with Encrypted SSH Keys

workflow git

This year, I went to FOSDEM 2024 . It was nice and ~~cosy~~ crowded, and I really enjoyed my time there. The very last talk I could attend to before having to leave for the train station was “So You Think You Know Git ” by Scott Chacon . If you haven’t already, go and watch it. It is a very good and educational presentation. You will learn what git blame -C -C -C does and never be the same.

Another takeaway for me was git maintenance. git maintenance allows running in the background a set of tasks which optimize commands like git add and git fetch for a responsive user experience. I mean, count me in! Our git repository at $WORK has become fairly big over the yearsNot mono-repo big yet, but still big enough to make git fetch --all --prune feels… well, unresponsive. .

So, when I came back to work the next day, I run the magic command the speaker had mentionedFollowing a petition from the Internet , my terminal prompt is ;. .

; git maintenance start

This created a bunch of user systemd services and timers which I decided to run immediately to test that everything was working correctly, starting with the hourly service responsible for prefetching remote branches.

; systemctl --user start git-maintenance@hourly.service

Unfortunately, this did not work out, and for predictable reasons.

; systemctl --user status git-maintenance@hourly.service
(...)
systemd[1706]: Starting Optimize Git repositories data...
git[76228]: git@gitlab.com: Permission denied (publickey).
git[76226]: error: failed to prefetch remotes
git[76226]: error: task 'prefetch' failed
systemd[1706]: git-maintenance@hourly.service: Main process exited, code=exited, status=>
systemd[1706]: git-maintenance@hourly.service: Failed with result 'exit-code'.
systemd[1706]: Failed to start Optimize Git repositories data.

The culprit here is the fact I am using an encrypted SSH key to connect to Gitlab where our repository is hosted and out of the box the scripts run by git-maintenance have now way to use them. This is because git-maintenance is not aware of the existence of the SSH agent running on my laptop.

The solution can be read in the Man page of git-maintenance:

(…) any customization should be done by creating a drop-in file, i.e. a .conf suffixed file in the ~/.config/systemd/user/git-maintenance@.service.d directory.

I didn’t know this general purpose trick which should work for any systemd service running on a Linux machine! Thanks, anonymous technical writer who took the time to contribute to this Man page.

And indeed, creating a file named 10-ssh.confThe prefix number is as important as the .conf suffix mentioned in the git-maintenance Man page for systemd to load the drop-in file. in ${HOME}/.config/systemd/user/git-maintenance@.service.d/ to set the SSH_AUTH_SOCK environment variable solved my issue. Its value depends on your personal setup. In my case, I am using the systemd ssh-agent.service user unit.

; systemctl --user show ssh-agent.service | grep Environment=SSH_AUTH_SOCK
Environment=SSH_AUTH_SOCK=/run/user/1000/ssh-agent.socket

We replicate this in our 10-ssh.conf.

[Service]
Environment=SSH_AUTH_SOCK=/run/user/1000/ssh-agent.socket

And we are done! This time, executing the service manually will work (assuming the necessary encrypted key has been ssh-add to the agent).

; systemctl --user daemon-reload
; systemctl --user start git-maintenance@hourly.service
; systemctl --user status git-maintenance@hourly.service
(...)
systemd[1706]: Starting Optimize Git repositories data...
systemd[1706]: Finished Optimize Git repositories data.

Building Static Binaries for OCaml Projects

December 31, 2023

Building Static Binaries for OCaml Projects

ocaml

Building static binaries can come in handy. Most notably, when the time comes to distribute executables. I should know, because I spend a bit of time recently preparing for the 6th release of Spatial Shell, and part of that time was ensuring that folks on Linux unwilling to build an OCaml project from source could still give my project a shot.

Turns out, it can be achieved pretty easily, when you know what you are doing. This is not the first article published on the Internet which explains how to build static binaries, but (1) the ones I have found were a bit verbose and hard to follow, and (2) the most recent versions of OCaml bring a new challenge to tackle.

Setting-Up a Dedicated `dune` Profile

At the root of your repository, create a dune file containing the following stanza:

(env
 (static
  (flags
   (:standard -cclib -static))))

This tells dune to passes specific arguments to the OCaml compiler when called with the --profile=static command-line argument.

At this point, calling dune build --profile=static is likely to fail with linking related errors. For instance, I got this one.

/sbin/ld: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/crt
beginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/sbin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status

With OCaml 5.1.0 and OCaml 5.1.1, you are also likely to hit an error related to zstd, which is a new runtime dependency for OCaml programsIt looks like a temporary issue. If I understood correctly, the OCaml core developers are planning to remove said dependency in future versions of the compiler. .

/sbin/ld: cannot find -lzstd: No such file or directory
collect2: error: ld returned 1 exit status

The solution for these two problems is the same.

Setting Up a Specific Opam Switch

The OCaml compiler has a lot of so-called variants. The most famous is probably flambda , which enables a series of optimisations for native compilation.

Nowadays, enabling the specific features of a given variant is achieved by installing the related option package when choosing your compiler version. In our case, we have to install:

ocaml-option-static (which will also enables ocaml-option-musl)
ocaml-option-no-compression (for OCaml 5.1.0 and 5.1.1)

The minimal Opam invocation to create such a switch is:

opam switch create . --packages "ocaml-option-static,ocaml-option-no-compression,ocaml.5.1.1"

This particular configuration will produce static binaries using musl , a lightweight libc implementation that is way more suited than the glibc when it comes to static binaries. This means you will have to install musl on your system (it is more than likely that it is packaged by your favorite distribution).

Additionally, you will have to provide the static versions (.a) of the system libraries your project relies on. This part might be tricky to address, and is very dependent on the particular system you are using. For instance, Debian makes this a lot easier than Archlinux. This was typically what was happening with zstd: Archlinux only distributed dynamic libraries, so no .a where available on my laptop. Another example: if your project depends on Zarith , then you have a dependency to gmp. You need to go and find the static library for it before moving to the next sectionOf course, it is also possible that you are working on a pure OCaml project whose only real dependency is to the libc. Lucky you. .

Verifying the Resulting Binaries

Now is the time to call Dune again, with the --profile=static command-line argument. Checking the result of your effort is as simple as calling ldd on the binary compiled by Dune.

    not a dynamic executable

If ldd outputs something of this form, then you are good to go!

Conclusion

This is, in a nutshell, the approach I am using to produce static binaries for Spatial Shell releases. You can find the resulting script in the project repository if you are interested.

If you have spotted an error in this article, or if you think the solution proposed here can be improved, do not hesitate to reach out to me (ideally though my public inbox). Or, even better, open an issue on Spatial Shell tracker ! Afterall, these are the very steps I’m relying on in order to provide static binaries, so I’d like to know if I am doing something wrong 😅.

Spatial Shell 6th Release Might Be the Charm

December 30, 2023

Spatial Shell 6th Release Might Be the Charm

spatial-shell releases

After a first call for testers that could have been more effective if only the building instructions listed in the README were correct , I am quite happy to announce the 6th release for Spatial Shell that, I believe, is pretty usable for someone who isn’t me.

What’s Spatial Shell Already?

In a nutshell, Spatial Shell implements a spatial model inspired by Material Shell for i3 and sway. Windows are organized in a grid whose rows are your workspace, and you can navigate this grid (by focusing neighbors windows), move the focused window around, decide how many window you want to see at most in every workspace, etc.

When I first discovered Material Shell in August 2022, I was pretty excited, but the fact that it is a GNOME extension prevented me from switching completely (besides, it was pretty unstable on my laptop for reasons. So when I discovered that Sway (my personal favorite window manager) could be controlled by a third-party program through a Unix socket , I started experimenting. It didn’t take long before I had a working prototype that fit most of my need , but it took a significant amount of time to turn this prototype into something really usable by anyone who isn’t me.

Here is a clunky showcase of what Spatial Shell can doYes, Scott Pilgrim Takes Off is an awesome show that you definitely need to watch. .

For the interested viewers, I am using a forked version of sway called SwayFX to get these rounded corners and the dim effects on unfocused windows. The status bar is Waybar with a config you can try yourself if you are interested ! There is also a minimal i3blocks config available if you cannot use WaybarI fully intend to provide an example config as visually pleasing as the one for Waybar, but compatible with i3. .

To be clear, the UI part of Spatial Shell remains lacking to this day. My Waybar configuration works, but is pretty verbose and statically limited. For instance, it will only display the 8th first windows of a workspace. This is because Waybar does not allow one widget to generate multiple blocks. The same goes for the i3blocks configuration example. Also, they assume the availability of a font with iconsOn that matter, I love the Nerd Fonts project. .

Try It Yourself!

As of today, you can install Spatial Shell using the following methods:

By fetching the official binary builds for Linux x86_64 from the GitHub releases page
By installing the AUR package for Archlinux
By building directly from source

The README should be a good starting point. If it is lacking key pieces of information, do not hesitate to open an issue. At this point, I am willing to deal with documentation issue as bugs.

Besides its README, Spatial Shell is documented via manpages. For a list of available manpages, you can refer to man -k spatial. Spatial Shell definitely works with sway, and should work with i3 since the former is basically a drop-in replacement for the latter.

Conclusion

The journey from the initial idea up to this moment has definitely been an interesting one. At this point, Spatial Shell provides every feature I need. That being said, it does not mean there is nothing more to tell! I certainly hope this project will catch the eyes of some of you, and soon be used on computers I do not own by folks who are not me.

Contributions are obviously welcome! Typically, if you happen to piece together a neat status bar configuration, I am very interested.

The Free and Open Source Software Projects This Website is Built Upon in June 2023

June 23, 2023

The Free and Open Source Software Projects This Website is Built Upon in June 2023

Authoring Contents

As mentioned in my latest retrospective, I have simplified how I write my contents. Nowadays, all my write-ups are written in Markdown, and I use markdown-it to parse them and generate nice and fancy HTML documents.

In addition to the base parser that markdown-it is, I am relying on an ever-growing collection of plugins:

As a consequence, this website also benefits from two very nice projects: highlight.js for syntax highlighting, and $\KaTeX$ for displaying mathematics equations.

Frontend

In addition to normalize.css to “reset” the CSS defaults of various browsers, this website could not exist as it is now without the awesome Tufte CSS project. The sidenotes of this website could not have existed without it.

HTML Generation

While markdown-it is doing the heavy lifting of turning my Markdown contents in HTML pages, soupault is the goldsmith which pieces these pages together into a coherent whole.

I really enjoy using soupault for my website. In a nutshell, this tool enables me to customize the pages of my website with an API very similar to what Javascript offers, but instead of the computation happening on the client side every time the page is loaded, it is done on my laptop once, just before I push the new pages. Just an example: for my sidenote pages, I was able to reuse the markdown-it-footnote with zero modification, and I just had to write a simple Lua plugin to customize its output to my (humble) needs.

The running log is also another good example of an ad hoc page I can generate using a simple Lua script.

Extensible Records in OCaml Using dmap

June 20, 2023

Extensible Records in OCaml Using `dmap`

ocaml

dmap is a library to create and manipulate heterogeneous maps. It features a very straightforward API which leverages the common trick of tagging the type of the keys with a parameter specifying the types of the associated valuesThis article assumes readers are familiar with GADTs. . That is, given 'a key the type of keys of a heterogeneous map t, then the value associated to a key k of type bool key is expected to be of type bool.

In this write-up, we show how it is possible to use dmap to implement extensible records (that is, records which can be extended with new fields after they have been defined) in OCaml. I have also published on GitHub a repository containing the implementation presented in this article . While dmap is far from being the only available solution for this use caseFor heterogeneous maps, gmap is apparently a popular alternative, and so is hmap . For extensible records, orec looks like a very interesting library I should probably take a long look at. , it is the one I happened to use for implementing a “plugin system” in a library where each plugin can manipulate a shared state in a type-safe way.

Encoding Extensible Records

The (simplified) module type of a heterogeneous map built using dmap is as follows.

module type S = sig
  type t
  
  type 'a key

  type binding = Binding : 'a key * 'a -> binding

  val empty : t

  val add : 'a key -> 'a -> t -> t

  val find_opt : 'a key -> t -> 'a option

  val fold_left : ('a -> binding -> a) -> 'a -> t -> 'a
end

The use case for heterogeneous maps I’ve been interested in recently is encoding records. More precisely, constructors of keys encode fields containing collections of values. In such approach, a constructor Foo : int key encodes a field of type int option, while a constructor Bar : string -> bool key encodes a field of type bool StringMap.t. That is, the heterogeneous map HMap.t, such as

type _ key = Foo : int key | Bar : string -> bool key

module HMap : S with type 'a key  = 'a key

can be used to encode values of a record type

type t = {
  foo : int option;
  bar : bool StringMap.t;
}

as the following conversion functions demonstrates.

let of_record {foo; bar} =
  let res = HMap.empty in
  let res = HMap.add Foo foo res in
  StringMap.fold_left
    (fun res (key, value) -> HMap.add (Bar key) value res)
    res
    bar

let of_hmap map =
  {
    foo = HMap.find_opt Foo map;
    bar = HMap.fold_left
            (fun smap (Binding binding) ->
               match binding with
               | (Bar k, v) -> StringMap.add k v smap
               | _ -> smap)
            StringMap.empty
            map;
  }

As a consequence, to construct a heterogeneous map whose type of keys is extensible is a way to get extensible records in OCaml.

To that end, dmap only asks for a function to compare two arbitrary keys. More precisely, given the type

type _ key = ..

the challenge is to write the function compare of type

val compare : 'a key -> 'b key -> ('a, 'b) Dmap.cmp

with Dmap.cmp being definedThis definition of cmp enforces that two keys can only be equal if and only if the values associated to them are of the same type. as

type (_, _) cmp =
  | Lt : ('a, 'b) cmp
  | Eq  : ('a, 'a) cmp
  | Gt : ('a, 'b) cmp

The Solution

To use extensible types in this kind of setting, the trick I am aware of is to use a registration mechanismFor instance, this is what the tezos-error-monad package is using for being able to print the values of its extensible error type . .

The compare function will work as follows.

We assign an integer to each constructor of the type key.
We compare these integers when the arguments of compare are constructed from different constructors.
We implement one comparison function per constructors, to be called when the two arguments of compare are the same.

The exercise is being made a bit more complicated than what the previous paragraph suggests due to the fact that both key and cmp are GADTs. As a consequence, the rest of the article will proceed as follows. First, we will implement the overall logic of the compare function implemented with a registration mechanism, and only then will we tackle the OCaml compiler errors GADTs so often bring.

Step 1: Forget About GADTs

To implement our registration mechanism, we rely on a good old reference to a listIt is a good idea to hide this implementation “details” behind a nice API, using a mli file. . The position of a registered object within the list will determine the integer attributed to it.

let registered_keys = ref []

let register_key key =
  registered_keys := key :: !registered_keys

In this list, we store a collection of first-class modules which provide us everything we need to compare two 'a key values obtained from the same constructors.

module type KEY = sig
  type t

  val proj : 'a key -> t option
  val compare : t -> t -> int
end

The type t of the module type KEY is the type of one constructor’s arguments. The simplest definition of t is unit, which means the field encoded with the related constructor points to a singleton value (as foo did in the previous section).

However, it is also possible to have keys’ constructors taking arguments, which would translate into a field pointing to collections of value. This was the case for bar, which was pointing to a mapping of strings to booleans.

The compare function will consist in iterating over the first-class modules registered in registered_keys, in order to determine to which ones the two arguments belong to.

This means that considering a call compare left right, for each first-class module of registered_keys, either

both left and right have not yet been associated to a first-class module,
or left has, but `right hasn’t
or right has, but left hasn’t
or both left and `right have been associated to a first-class module.

Only when the fourth stage has been reached can we provide the expected result of compare, whose type we have simplified in this section to return a int instead of the GADT Dmap.cmp.

To encode these stages, we introduce a dedicated accumulator type.

type ('a, 'b) acc =
  | Init : 'a key * 'b key -> ('a, 'b) acc
  | Compare_left_with : 'a key * int -> ('a, 'b) acc
  | Compare_right_with : int * 'b key -> ('a, 'b) acc
  | Res : int -> ('a, 'b) acc

This allows us to implement compareNote the use of the Seq module here. It is motivated by the fact that List.fold_lefti does not exist. We need to know the position of the current first-class module we are testing the keys against, hence the need for a fold_lefti. Of course, we could have added the current position to the acc type, but the function is already cumbersome enough, and using Seq.t is mostly free. .

let compare : type a b. a key -> b key -> int =
 fun left right ->
  List.to_seq !registered_keys
  |> Seq.fold_lefti
       (fun (acc : (a, b) acc) i (module K : KEY) ->
         match acc with
         | Init (left, right) -> (
             match (K.proj left, K.proj right) with
             | Some left, Some right -> Res (K.compare left right)
             | Some _, None -> Compare_right_with (i, right)
             | None, Some _ -> Compare_left_with (left, i)
             | None, None -> acc)
         | Compare_right_with (j, right) -> (
             match K.proj right with
             | Some _ -> Res (Int.compare j i)
             | None -> acc)
         | Compare_left_with (left, j) -> (
             match K.proj left with
             | Some _ -> Res (Int.compare i j)
             | None -> acc)
         | _ -> acc)
       (Init (left, right))
  |> function
  | Res x -> x
  | _ ->
      raise
        (Invalid_argument
           "comparison with at least one unregistered key variant")

This function gives us the overall structure and logic of compare, but of course we are not done yet.

Step 2: Deal with GADTs

In practice and as hinted in the introduction of this section, dmap is not expected a comparison function of type 'a key -> 'b key -> ('a, 'b) Dmap.cmp, not 'a key -> 'b key -> int.

The first step is to update acc accordingly.

type ('a, 'b) acc =
   | Init : 'a key * 'b key -> ('a, 'b) acc
   | Compare_left_with : 'a key * int -> ('a, 'b) acc
   | Compare_right_with : int * 'b key -> ('a, 'b) acc
-  | Res : int -> ('a, 'b) acc
+  | Res : ('a, 'b) Dmap.cmp -> ('a, 'b) acc

Then, we focus our attention to modifying compare, by modifying its type signature, and by updating its definition to make use of the new acc type.

-let compare : type a b. a key -> b key -> int =
+let compare : type a b. a key -> b key -> (a, b) Dmap.cmp =
  fun left right ->
   List.to_seq !registered_keys
   |> Seq.fold_lefti
       (fun (acc : (a, b) acc) i (module K : KEY) ->
          match acc with
          | Init (left, right) -> (
              match (K.proj left, K.proj right) with
-             | Some left, Some right -> Res (K.compare left right)
+             | Some left, Some right ->
+                 let x = K.compare left right in
+                 Res (if x = 0 then Eq else if x < 0 then Lt else Gt)
              | Some _, None -> Compare_right_with (i, right)
              | None, Some _ -> Compare_left_with (left, i)
              | None, None -> acc)
          | Compare_right_with (j, right) -> (
              match K.proj right with
-             | Some _ -> Res (Int.compare j i)
+             | Some _ ->
+                 let x = Int.compare j i in
+                 Res (if x < 0 then Lt else Gt)
              | None -> acc)
          | Compare_left_with (left, j) -> (
              match K.proj left with
-             | Some _ -> Res (Int.compare i j)
+             | Some _ ->
+                 let x = Int.compare j i in
+                 Res (if x < 0 then Lt else Gt)
              | None -> acc)
          | _ -> acc)
        (Init (left, right))

At this point, we are remembered that the OCaml compiler is not that easy to please when GADTs are involved. In particular, it complains for our use of Eq.

46 |               Res (if x = 0 then Eq else if x < 0 then Lt else Gt)
                                      ^^
Error: This expression has type (a, a) Dmap.cmp
       but an expression was expected of type (a, b) Dmap.cmp
       Type a is not compatible with type b

This error message is the result of a tragedy in two acts:

The Eq constructor of cmp is of type ('a, 'a) cmp. That is, only two keys associated with values of the same type can be equal.
compare takes two keys whose types can be different, and even if proj is expected to return a Some only when both keys are constructed with the same constructor, nothing in its type enforces that invariant.

As such, OCaml is not convinced with our code alone that a = b, and rejects it.

One possible trick here is to rely on an equality witness typeThis is the trick to keep in mind when playing with GADTs, to a point where eq could arguably be added to the OCaml’s standard library. , and refine proj in order to convince OCaml our code is type-safe.

type (_, _) eq = Refl : ('a, 'a) eq

We then modify KEY to have proj not only return the arguments of the constructor, but also a type equality witness to unify the polymorphic argument of key to the expected value type of said constructor.

 module type KEY = sig
   type t
+  type r
 
-  val proj : 'a key -> t option
+  val proj : 'a key -> (t * ('a, r) eq) option
   val compare : t -> t -> int
 end

This is enough to convince OCaml that a = b, because it gets a proof that a = K.r and b = K.rBut only if you actually match Refl explicitly, meaning the pattern Some (left, _), Some (right, _) would still lead to the same error as not returning Refl at all. .

           match (K.proj left, K.proj right) with
-          | Some left, Some right ->
+          | Some (left, Refl), Some (right, Refl) ->
               let x = K.compare left right in
               Res (if x = 0 then Eq else if x < 0 then Lt else Gt)
           | Some _, None -> Compare_right_with (i, right)
           | None, Some _ -> Compare_left_with (left, i)

That’s it. compare is done, compiles, and satisfies the expectations of dmap. We can instantiate its functor with our key extensible typeThis approach will lead every value of type Extensible_record.t to share the same fields, which might not be you need. One can imagine embedding the registered_keys list of known keys inside the extensible records values, along with a heterogeneous map. This would allow each value to have its own dynamic set of fields. .

module Extensible_record = Dmap.Make (struct
  type 'a t = 'a key

  let compare = compare
end)

Step 3: Profit

To wrap up this article, we can come back to our initial example of a record with two fields foo and bar.

Here is how we can extend key to have a Foo constructor.

type _ key += Foo : int key

let () =
  register_key
    (module struct
      type t = unit
      type r = int

      let proj : type a. a key -> (t * (a, r) eq) option = function
        | Foo -> Some ((), Refl)
        | _ -> None

      let compare () () = 0
    end)

Similarly, here is how we can register Bar.

type _ key += Bar : string -> bool key

let () =
  register_key
    (module struct
      type t = string
      type r = bool

      let proj : type a. a key -> (t * (a, r) eq) option = function
        | Bar arg -> Some (arg, Refl)
        | _ -> None

      let compare = String.compare
    end)

Once registered, these two fields/keys can be used to populate an Extensible_record.t value with values of the expected types.

let () =
  let record = Extensible_record.empty in
  let record = Extensible_record.add Foo 3 record in
  let record = Extensible_record.add (Bar "foobar") true record in
  assert (Extensible_record.find Foo record = 3) ;
  assert (Extensible_record.find (Bar "foobar") record)

A nice happy consequence of this approach is that it allows us to register private fields only a given module can manipulate. Indeed, if this module does not expose the constructor of the key it relies on, then only it can interact with this part of the extensible record.

The Chaotic Debut of My Software Projects

May 29, 2023

The Chaotic Debut of My Software Projects

opinions

I am no stranger to the exciting feeling of starting new projects. By dint of repeating this “exercise” over the years, I have come to build an intuitive process which works for me. In this write-up, I want to give a try at describing it, both for future references and in the hope that this might start interesting conversations. I am curious to know whether or not my personal “process” is shared with some of my fellow developers, though I would be surprised if it weren’t the case, since there is nothing particularly clever or revolutionary with what I am about to write.

A Story in Two Acts

In a nutshell, I can often divide the early days of my software projects into two phases: the exploration phase and the rationalization phase.

The starting point of the exploration phase is a need, and a direction I want to take to address this need. For instance, the starting point of the Compact module of data-encoding was the need to encode in bytes arbitrary values, and the direction was to do that with data-encoding. For Spatial Shell , I wanted badly to enjoy Material Shell ’s user experience without having to switch to Gnome 3 and the direction was set when I discovered sway’s RPC protocol .

The goal of the exploration phase is for me to turn the direction into a destination, so to speak. When I start, my knowledge of both the problem I want to solve and the solution I am trying to build is vague, and needs refinement. There are intuitions I need to challenge, APIs I need to learn, boilerplate I need to understand. More often than not, this phase is messy. Commits are not nice, standalone units of changes: it is way too early for that. I use Git merely as a way to throw away a dead-end, and resume exploring down another path from a reasonable save point. I put most if not all my code into one file, I don’t write comments, encapsulation and abstraction become foreign words I forget about for a while. My goal is to come up with a prototype which captures the key features I am looking for, and to learn as much as possible in the process. This exploration phase shouldn’t last too long, because it is very context sensitive. If I am interrupted long enough, I am likely to lose the mental map of my messy code, and to forget the key insights I was learning while navigating it.

I want to emphasis that the exploration phase should definitely touch to the mundane, boring things like the CLI, the decoding of configuration files, the encoding of persistent states, etc. These things tend to have a pervasive impact on the rest of a software development if they are not taken into account early. Because I am confronting myself to these forbidding parts, I am forced to understand the pain points they will bring, and to plan for them.

At some point, most of my questions have found their answers, and I am starting to tackle features which go beyond the scope of the minimal set of features I was initially after. When this time comes, friction is starting to appear because of the bad quality of the codebase. All the shortcuts are coming back to bite me and slow me down, while wild bugs enjoy the lack of invariant enforcement and multiply.

This is actually a good sign: the exploration is over, and before going any further, it is time for me to take a step back and rationalize.

The rationalization phase is the unique opportunity to “learn what I have learned,” by revisiting every line of the code you have written, challenge it, and hopefully come up with a solution for its most outstanding issues. I do that by splitting up the monolithic that came out of the exploration phase into a sane, navigable hierarchy of modules with clear, manageable APIs. This allows me to fine-tune the design, to tweak the parts of the API exposed to the users (the “boring parts” I mention for the exploration phase), and to remove any dead code I thought might be useful or needed, but actually didn't cut it to the prototype.

This second phase often feels like a sprint. The second I am throwing myself at it, the project stops working, and most of the time, it won’t even fully compile before I’m actually done. This sometimes feels a bit overwhelming, though the use of strongly typed programming language like OCaml or Rust has proven to be of tremendous help, at least as long as I stick to moving code around, and only amending it at the margin. Clearly, types and compile-time type checking are my allies during what I often experience as a long snorkeling session.

When I’m done, I get my prototype working again, except it does not feel like a prototype anymore. The feature set is the same, as is the user experience. But the developer experience is totally different. I can now enjoy a codebase where separation of concerns, encapsulation and abstraction play their rightful roles, and allow me to reason about my software project more easily. This is also the moment where I can slow down a bit, since it is no longer required for me to keep the whole project architecture in my head. To a large extent, the module hierarchy does that for me.

Random Thoughts

To me, this process feels a bit like a craft. I think I have become better at going through it after every project I successfully bring at the end of the rationalization phase. And I often feel a bit at lost when I am involved in a project which does not follow this general approach.

That being said, I have to admit it does not scale very well. If the whole process does not fit in something like a week, then I will likely run out of motivation before reaching stable grounds. For instance, the rationalization phase often one to two days, during which I am basically stuck to my computer for long hours.

Finally, this process does not leave a lot of room for collaborating with other developers. This is true both for the exploration phase (the “codebase” is still a monolithic file that is partially rewritten over and over) and for the rationalization phase (sometimes it feels like I am “discovering” the correct way to organize the codebase while I organize it). However, collaboration becomes possible (and even very rewarding) after the fact, when the codebase “makes sense.”

I guess it depends on the tradeoffs one is willing to make when starting a project, the size and scale of the project, and the software developers and entities involved.

What happened since December 2022?

May 18, 2023

What happened since December 2022?

emacs meta neovim releases spatial-shell

Initially, I started this “What happened” series as an exercise to publish more regularly on this website. Suffice to say, I haven’t done a particularly impressive job in that regard, which only means I have a lot of room for improvement.

Anyway, if the first few months of 2023 has been mostly $WORK focus, the same cannot be said for April and May. For one, I have started running again. But this is only the tip of the iceberg.

Spatial Shell got its first releases

Spatial Shell is probably my hobby project I am most excited about. The “call for testers” article I have published recently managed to catch the attention of a few folksYou want to hear a lesson I learned the hard way just after publishing it? Before calling for testers, it is better to be sure your project can actually be compiled easily by the potential volunteers . . The perspective to publish such a write-up was a very strong source of motivation for me to clean up a project I was using daily for several months now, and I am very satisfied with the result.

Mass adoption is still a distant horizon, but still, the project is now mainstream enough that it has already been mentioned in a random topic on the OCaml discourse by someone who isn’t me . 🎉

This led me to formally release a first version of Spatial Shell in the end of April, and a second today. For the first time, I have also published an Archlinux package , to make the life of potential early adopters even easier. Do not hesitate to upvote it so that it can find its way to the extra repository some day.

Goodbye Emacs! Remember me, Neovim?

In 2015, I started using Coq for my PhD thesis and at the time, there was no real support for (Neo)vimThe situation later improved. Nowadays, you can implement your theories using Coqtail , and Coq LSP will probably become a viable and interesting setup in a near future. . Everyone was using Proof General and Emacs, so I was left with little choice but to follow through. With only my courage and the good advice of a fellow “vimer” who had also made a similar journey , I started using Emacs.

Fast forward 8 years later, and my Emacs configuration has become a project of its own. Overall, I was pretty happy with my setup, but in the same time, I always remained a bit nostalgic of my Neovim days. This is probably why I decided to give this old friend a try when my company bought me a new laptop. I also used this as an opportunity to try out this LSP-thing everyone was talking about.

It has been a month now, and I do not plan to come back to my previous habits. There are still some few edges here and there, I still need to get my head around lua, but LSP is nice, and plugins like telescope are simply too beautiful.

That being said, there was one aspect of moving from Emacs to Neovim I had not anticipated: Org mode. Which constitutes a perfect transition to the next session.

Website redesign, again

Did you notice this website has been revamping recently? The changes are actually deeper than “just” a redesign, to a point where I had to port all my write-ups to a different markup languageAre you starting to understand why “Org mode” was the perfect transition to move on to this section? .

Why, you ask? Well, it’s actually pretty simple: as time goes, I’ve grown lazier.

Let me give you some context. Until very recently, my website was built around the idea to have literate programming as a first-class citizen of my author tools. For instance, you can have a look at what used to be the literate program which was responsible for generating the website. Similarly, most of my write-ups about Coq were actually Coq documents. Literate programming is actually a very nice paradigm for authoring technical contents, because it gives you the tools to keep said contents accurate and up-to-date. In a nutshell, you cannot have a typo in one of your code snippets which would prevent it from compiling, because you actually compile the snippet and catch the typo when you try to generate your website. Or at least, it is what I used to do.

I decided to stop because, for all its benefits, this approach has one major drawback: it is hard to maintain. I had invested quite some time and efforts to keep my website sources under control, but it really was an everyday fight. There are some strange things which start happening when you fully commit to this, as I think I did. For instance, software dependencies tie your article together. Suddenly, you cannot talk about this new fancy feature of the latest Coq release without upgrading all your write-ups implemented as Coq documentsWell, in theory you can. Just have each Coq document specifies the Coq version it requires, and support this level of customization in your build toolchain. But then, your blog takes forever to build from a cold repository. .

That being said, most of the work had already been done. This website was a collection of literate programs, and I was pretty proud of the state of things. I could deal with the annoyancesLike using Coqdoc syntax to write my articles, for instance. I could write about how the Coqdoc syntax irks me for ages. . But then, as I explained in the previous section, I decided to move away from Emacs. The first time I tried to start a new write-up, it hit me.

I used to write most of my contents using Org mode. Org mode, also known as the Emacs markup language.

I know of at least one “Org plugin” for Neovim , but instead of giving it a try, I decided to use this opportunity to tackle my “maintenance problem” once and for all. I gave up on literate programming for this website. As a result, this website is now generated from Markdown files only (using markdown-it with many plugins). As a consequence, the generated HTML is way more “predictable.” This was enough to motivate me at giving a try at Soupault’s indexes , which are way more powerful than I anticipated. Now, this website has

Tags. Each write-up can be labeled with as many tags as I want, there is a page which lists all the tags used in the website, and each tag has its own page (for instance, the coq tag.
A RSS feed. It was actually one of the main features I really wanted to get with this revamp.
Automatically generated list of articles in the home page, for each series (see the Ltac series for instance). Before, I was publishing “curated indexes,” or put in other words: I was writing these indexes myself, by hand. And again, I’ve grown lazier.

It took me a week to go through this rework. Translating manually every write-up was tedious, to say the least, as was implementing the Lua plugins for Soupault since I have neither proficiency nor tooling to help me write Lua code. But I am very glad for the final result.

Also, I have invested in an Antidote license, so hopefully, this website will have fewer typos and English butchering as of now. A clean text, delivered with a nice and simple design, from a sane and maintainable Git repository .

Neovim, OCaml Interfaces, Tree-Sitter and LSP

May 1, 2023

Neovim, OCaml Interfaces, Tree-Sitter and LSP

ocaml neovim

Can we all agree that witnessing syntax highlighting being absolutely off is probably the most annoying thing that can happen to anybody?

I mean, just look at this horror.

Syntax highlighting being absolutely wrong.

What you are looking at is the result of trying to enable tree-sitter for OCaml hacking and calling it a day. In a nutshell, OCaml mli files are quickly turning into a random mess of nonsensical colors, and I didn’t know why. I tried to blame tree-sitter-ocaml , but, of course I was wrong.

The issue is subtle, and to be honest, I don’t know if I totally grasp it. But from my rough understanding, it breaks down as follows.

tree-sitter-ocaml defines two grammars: ocaml for the ml files, and ocaml_interface (but ocamlinterface also works) for the mli files
By default, neovim uses the filetype ocaml for mli files, so the incorrect parser is being used for syntax highlighting. This explains the root issue
Bonus: ocamllsp does not recognize the ocamlinterface filetype by default (but somehow use the ocaml.interface id for mli files…There is probably something to be done here. ).

So, in order to have both tree-sitter and ocamllsp working at the same time, I had to tweak my configuration a little bit.

lspconfig.ocamllsp.setup({
  filetypes = vim.list_extend(
    require('lspconfig.server_configurations.ocamllsp')
      .default_config
      .filetypes,
    { 'ocamlinterface' }
  ),
})

vim.cmd([[au! BufNewFile,BufRead *.mli setfiletype ocamlinterface]])

And now, I am blessed with a consistent syntax highlighting for my mli files.

Syntax highlighting being absolutely right.

Spatial Shell: Call For Testers

April 27, 2023

Spatial Shell: Call For Testers

spatial-shell

In August 2022, I have discovered Material Shell . A few weeks later, I had pieced together a working prototype of a dynamic tiling management “a la Material Shell” for sway . By October, the project was basically fulfilling my needs, and I had already started to use it on my workstationI tried so you do not have to: having my graphical session going crazy during a work meeting because of a software I had written. . The project sat there for a while until I rediscovered this thing called holidays.

For a short week, I tried to address at many of the remaining issues and missing features that I was aware of. Then, I started to write man pages , which turned out to be the perfect opportunity to clean up every clunkiness I could possibly find.

I can’t help but find the result rather nice and satisfying, and I hope you will enjoy it too! Spatial Shell works on my machine, which means it will definitely break on yours. But this is where the fun lies, right? At this point, I definitely think the project is ready to fall into the hands of (motivated) alpha testers.

Anyway, let me give you a tour!

Spatial Model

At its core, Spatial Shell allows you to navigate a grid of windows. Workspace are rows which can be individually configured to determine how many windows (cells) you can see at once. More precisely, workspaces in Spatial Shell can use two layouts:

Maximize: One window is displayed at a time
Column: Several windows are displayed side by side, to your convenience

The reason Maximize is not a particular case of Column, but instead a layout on its own, is to easily allow you to switch to and back from maximizing the focused window. The following pictureCreated using Excalidraw . summarizes one particular setup with tree workspaces, each configured differently.

Spatial Shell allows users to configure the layout of each workspace individually.

Workspace 1 contains three windows, and uses the Column layout to display at most three windows, so each window is visible, with the focus being on the leftmost one.
Workspace 2 contains four windows, and uses the Column layout to display at most two windows. As a consequence, two windows are not visible.
Workspace 3 contains two windows, and uses the Maximize layout so only the focused window is visible.

To help users know which window is currently holding the focus, Spatial Shell slightly reduces the opacity of unfocused windows (as poorly hinted by the gray backgrounds in the figure). Finally, Spatial Shell can also set a background for empty workspaces (using swaybg under the hood).

And this is basically it. There is not much more to say about Spatial Shell features.

Configuration

From an implementation and user experience perspective, Spatial Shell is taking inspiration from i3 and sway.

More precisely, Spatial Shell comprises two executables:

spatial(1) , the daemon implementing the spatial model described in the previous section, and
spatialmsg(1) , a client used to control the current instance of spatial.

Assuming $spatial and $spatialmsg contains the paths to spatial and spatialmsg binaries respectively, then the simplest sway configuration to start using Spatial Shell is the following

exec $spatial

# moving the focus
bindsym $mod+h exec $spatialmsg "focus left"
bindsym $mod+l exec $spatialmsg "focus right"
bindsym $mod+k exec $spatialmsg "focus up"
bindsym $mod+j exec $spatialmsg "focus down"

# moving the focused window
bindsym $mod+Shift+h exec $spatialmsg "move left"
bindsym $mod+Shift+l exec $spatialmsg "move right"
bindsym $mod+Shift+k exec $spatialmsg "move up"
bindsym $mod+Shift+j exec $spatialmsg "move down"

# toggle between Maximize and Column layout
bindsym $mod+space exec $spatialmsg "toggle layout"

# modify the number of windows to display in the Column layout
bindsym $mod+i exec $spatialmsg "column count decrement"
bindsym $mod+o exec $spatialmsg "column count increment"

# start waybar, spatial will take care of the rest
exec waybar

By default, Spatial Shell sets the initial configuration of a workspace to the Column layout with two columns at most, and sets the opacity of the unfocused windows to 75%. This can be customized, either globally or per workspace, by creating a configuration file in $XDG_CONFIG_HOME/spatial/configIf unset, $XDG_CONFIG_HOME defaults to $HOME/.config. .

For instance, my config file looks like that.

[workspace=3] default layout maximize
background "~/.config/spatial/wallpaper.jpg"
unfocus opacity 85

See spatial(5) for the list of commands supported by Spatial Shell.

As a side note, readers familiar with sway will definitely pick the resemblance with sway and swaymsg, and it actually goes pretty deep. In a nutshell, swaymsg connects to a UNIX socket created by sway at startup time, to send it commands (see spatial(5) ) using a dedicated IPC protocol inherited from i3 (see sway-ipc(7) ). Not only spatial also relies on sway IPC protocol to interact with sway and implement its spatial model, it creates a UNIX of its own, and supports its own protocol (spatial-ipc(7) ).

Waybar Integration

It is a common practice to use a so-called “bar” with sway, to display some useful information about the current state of the system. In the contrib/ directory of Spatial Shell repository , interested readers will find a configuration for Waybar Readers familiar with Material Shell design will not be surprised by the general look and feel of the screenshot below. . This configuration is somewhat clunky at the moment, due to the limitations of the custom widget of Waybar which does not allow to have one widget defines several “buttons.” I am interested in investing a bit of time to see if I could write a native widget, similarly to sway’s one.

Mandatory screenshot of Spatial Shell, with the Waybar configuration.

That being said, the user experience with this integration is already pretty neat. As long as you don’t need more than 6 workspaces and 8 windows per workspacesThese constants are totally arbitrary and can be increased by modifying the Waybar config, but the issue will remain that a limit will exist. , you are good to go!

Building from Source

As of April 2023, the only way to get Spatial Shell is to build it from source.

You will need the following runtime dependencies:

sway (i3 might be supported at some point)
gmp (for bad reasons, fortunalety this will be removed at some point)
swaybg
waybar (if you want the full experience)

You will need the following build dependencies:

opam
scdoc (for the man pages)

Then, building and installing Spatial Shell is as simple as using the two following commands.

make build-deps
make install

The latter command will install Spatial Shell’s binaries in /usr/local/bin, and the man pages in /usr/local/man. You can remove them with make uninstall.

To install Waybar theme, copy contrib/waybar/spatialbar.py to /usr/local/bin/spatialbar for instance, and the Waybar style and config file to $HOME/.config/waybar.

Patch Dependencies for Stacked Git

January 26, 2023

Patch Dependencies for Stacked Git

git opinions

Every time I catch myself thinking about dependencies between changeset of a software project, the fascinating field of patch theories comes to my mind.

A “patch theory” usually refers to the mathematical foundation behind the data model of so-called Patch-based DVCS like Darcs and Pijul. More precisely, a patch theory is an encoding of the state of a repository, equipped with operations (gathered in so-called patches, not to be confused with GNU diff patches) one can do to this state. For instance, my rough understanding of Pijul’s patch theory is that a repository is an oriented graph of lines, and a patch is a set of operations onto this graph.

An interesting aspect of patch theory is that it requires a partial order for its patches, from which a Patch-based DVCS derives a dependency graph. In a nutshell, a patch $P$ depends on the patches which are responsible for the presence of the lines that $P$ modifies.

I have always found this idea of a dependency graph for the patches of a repository fascinating, because I though it would be a very valuable tool in the context of software development.

I wanted to slightly change the definition of what a patch dependency is, though. See, the partial order of Darcs and Pijul focus on syntactic dependencies: the relation between lines in text files. They need that to reconstruct these text files in the file system of their users. As a software developers writing these text files, I quickly realized these dependencies were of little interest to me, though. What I wanted to be able to express was that a feature introduced by a patch $P$ relied on a fix introduced by a patch $P'$ .

I have experimented with Darcs and Pijul quite a bit, with this idea stuck in the back of my mind. At the end of this journey, I convinced myselfI am not trying to convince you, per say. This is a very personal and subjective feedback, it does not mean someone else couldn't reach a different conclusion. (1) this beautiful idea I had simply could not scale, and (2) neither I nor our industry is ready to give up on the extensive ecosystem that has been built on top of git just yet. As a consequence, my interest in Patch-based DVCS decreased sensibly.

Until very recently, that is. I got reminded of the appeal of a dependency graph for changesets when I started adopted a Gitlab workflow centered around Stacked Git and smaller, sometimes interdependent MRs.

A manually curated graph dependency for a whole repository is not practical, but what about my local queue of patches, patiently waiting to be integrated into the upstream repository I am contributing too? Not only does it look like a more approachable task, it could make synchronizing my stacked MRs a lot easier.

The workflow I have in mind would proceed as follows.

Stacked Git’s new and edit commands could be extended to let developers declare dependencies between their patches. It would be the commands’ responsibility to enforce the wellfoundness of the dependency graph (e.g., prevent the introduction of cycles in the graph, and maybe diamonds tooAt least in a first version. There is definitely value in being able to work with two independent patches in conjunction with a third one that needs them both. That being said, our goal here is to organize our work locally, and if it is made easier by declaring artificial dependency, this is a pragmatic sacrifice I am personally willing to make. ).
The series command could be improved to display the resulting dependency graph.
push and pop would automatically take care (pushing or popping) of the selected patch(es) dependencies.
Ideally, Stacked Git would get a new command prepare <PATCH NAME> which would pop every patches applied, then only only push <PATCH NAME> and its dependencies (in the reverse order). Developers could fix conflicts if need be. That is, Stacked Git would not be responsible for the consistency or correctness of the dependency graph.
Stacked Git could get commands to detect potential issues with the dependency graph specified by the developer (mostly consisting in dry-run of prepare to check if it would lead to conflicts).

Because what we want is semantic dependencies, not syntactic dependencies between patches, I really think it makes a lot of sense to completely delegate the dependencies declaration to the developerFurther versions of Stacked Git could explore computing the dependency graph automatically, similarly to what Git does. But I think that if Darcs and Pijul told us anything, it's that this computation is far from being trivial. . The very mundane example that convinced me is the CHANGELOG file any mature software project ends up maintaining. If the contribution guidelines require to modify the CHANGELOG file in the same commit as a feature is introduced, then the patches to two independent features will systematically conflict. This does not mean, from my patch queue perspective, I should be forced to pop the first commit before starting to work on the second one. It just means that when I call stg prepare, I can have to fix a conflict, but fixing Git conflicts is part of the job after allAnd we have tools to help us. I wonder to which extends git rerere could save the day in some cases, for instance. . If for some reasons solving a conflict proves to be too cumbersome, I can always acknowledge that, and declare a new dependency to the appropriate patch. It only means I and my reviewers will be constrained a bit more than expected when dealing with my stack of MRs.

I am under the impression that this model extends quite nicely the current way Stacked Git is working. To its core, it extends its data model to constraint a bit push and pop, and empowers developers to organize a bit its local mess.

How I Keep Using Stacked Git at $WORK

January 16, 2023

How I Keep Using Stacked Git at `$WORK`

git workflow

One year ago, I have published an article summarizing my experience using Stacked Git at $WORK. Twelve months later, enough has changed to motivate a spin-off piece.

Stacked Git is Fast

Firstly, it is important to state that my main complaint about Stacked Git is now a thing of the pastFor fairness, I have removed the related section in my previous write-up. ! Stacked Git does not feel slow anymore, and far from it. This is because Stacked Git 2.0 has been rewritten in Rust . While RiiR (Rewrite it in Rust) is a running meme on the Internet, in this particular case, the result is very exciting.

Thanks to the performance boost, my Zsh prompt does not take 0.1s to appear!

Speaking of Zsh prompt, basically what I ended up displaying is (<TOP PATCH NAME> <APPLIED PATCHES COUNT>/<PATCHSET SIZE> <HIDDEN PATCHES COUNT). For instance, (fix-1337 1/2 3).

In case you want to take inspiration in my somewhat working configuration, here is the snippet of interest.

local series_top="$(stg top 2> /dev/null)"
local total="$(stg series 2> /dev/null | wc -l)"
local hidden="$(stg series --hidden 2> /dev/null | wc -l)"

if [[ "${total}" -gt 0 ]]; then
    local not_applied="$(stg series | grep -E '^-' | wc -l)"
    local applied="$(($total - $not_applied))"

    if [[ -z "${series_top}" ]]; then
        series_top="·"
    fi

    echo -n "(${status_color}${series_top} ${applied}/${total} ${hidden})"
    echo -n "  ($(current_branch))"
fi

Branchless Workflow

Last year, I was using Stacked Git on top of git branches. More precisely, I had one branch for each (stack of) Merge Request. It worked well, because my typical MR counted 3 to 4 commits in average.

Fast forward today, and things have changed on this front too. In a nutshell, I have become a “one commit per MR” maximalist of sortIt goes without saying that this approach comes with its set of drawbacks too. During the past year, I’ve pushed fairly large commits which could have been split into several smaller ones, for the sake of keeping my “one commit per MR” policy. I have also had to manage large stacks of MRs. . I find this approach very effective to get more focused reviews, and to reduce the time it takes for a given MR to be integrated into the main branch.

My previous approach based on git branches did not scale well with this new mindset, and during the course of the year, I stopped using branches altogetherI have not invented the branchless workflow, of course. After it was first published, someone posted a link to my Stacked Git article on Hacker News, and @arxanas posted a comment about git-branchless . I tried the tool, and even if it never clicked for me, I was really compelled by its core ideas. Similarly, Drew DeVault has published a complete article on its own branchless workflow in 2020 . .

These days, I proceed as follows.

I name each patch after the branch to which I will push it on our upstream Git remote.
99% of the time, I push my work using git push -f upstream @:$(stg top)
I created a small git plugin I called git-prepare which allows me to select one of the patch of my current patchset using fzf, and which pops all other patches that are currently applied.

git-prepare is really straightforward:

#!/bin/sh
patch=$(stg series -P | fzf)

if [[ ! $? -eq 0 ]] ; then
    exit $?
fi

if [ -n "$(stg series -A)" ]; then
    stg pop -a
fi

stg push ${patch}

The main hurdle which I still need to figure out is how to deal with stacked MRs. Currently, this is very manual. I need to remember which commit belongs to the stack, the order and dependencies of these commits, and I need to publish each commit individually using stg push; git push @:$(stg top).

The pragmatic answer is definitely to come back to git branches for this particular use case, but it's not the fun answer. So from time to time, I try to experiment with alternative approaches. My current intuition is that, by adopting a naming convention for my patches, I could probably implement a thin tooling on top of Stacked Git to deal with dependents commits.

Conclusion

Putting aside stacked MRs for now, I am really satisfied with my workflow. It’s very lightweight and intuitive, and working without Stacked Git now feels backward and clunky.

So I will take this opportunity to thank one more time Stacked Git’s authors and contributors. You all are making my professional like easier with your project.

What happened in October and November 2022?

November 19, 2022

What happened in October and November 2022?

spatial-shell nanowrimo coqffi

It is November 19 today, and I’m one month and 4 days late for the October Retrospective! Truth is, $WORK has been intense lately, to a point where I have not made much progress on my side projects. Anyway.

I have implemented the last feature I was really missing in my daily use of Spatial Sway: moving windows to adjacent workspaces. As a result, I think I can say that Spatial Sway has really reached the “Minimum Viable Product” stage, with a convenient UX, and a nice enough UI. It is still lacking when it comes to configurability, though. It is the next item of my TODO list, but I have no idea when I will implement the support for a configuration file.

Another highlight of the past two months was the NaNoWriMo . I took the last week of October and the first week of November off to plan and start writing a fiction project for it. Writing again was really nice, and I even gave writing fiction in English a shot. That made me uncover a bug in the English support of ogam , my markup language for fiction writers, which led me to publish a fix on Crates.io. However, as soon as I came back to $WORK, my writing spree ended. That’s okay, though. It gave me plenty of ideas for future sessions. Thanks, NaNoWriMo! Sorry to quit so soon, and see you next year, maybe.

Finally, a nice surprise of the past month is that someone has started working on adding proper support for coqffi to dune , the build system for OCaml and Coq! I’m thrilled by this. Thanks, @Alizter !

This wraps up this retrospective. I hope I will have more interesting, concrete news to share next month.

What happened in September 2022?

September 18, 2022

What happened in September 2022?

spatial-shell meta

It is September 18 today, and it has already been a month since I decided to start these retrospectives. This means it is time to take a step back and reflect of what happened these past few thirty days or soThere is the shocking news that I have started to use syntax highlighting again. But let’s not linger too much into it just yet. .

Spatial Sway

A few days after publishing my August Retrospective, I have learned the existence of Material Shell , an extension for GNOME 3 that provides a very interesting user experience.

I tried it for a few hours, but the thing kept crashing (it’s probably on me, I did not even remember I had Gnome installed on my machine, and I would not be surprised the culprit was my dusty setup rather than Material Shell itself). The experience remained very promising, though. Their “spatial model” especially felt like a very good fit for me. Basically, the main idea is that you have a grid of windows, with your workspaces acting as the rows. You can navigate horizontally (from one workspace to another), or horizontally, and you choose how many windows you want to see at once on your screen.

And so for a few hours, I was a bit frustrated by the situation… until I learned about how one can actually manage and extend Sway (the Wayland compositor I use for several years now) thanks to its IPC protocol. I spend like three days experimenting, first in Rust, then in OCamlThis was actually an interesting thought process. I am using OCaml at $WORK for about more than a year now. I have curated a setup that works pretty well, and I am familiar with the development tools. On the contrary, I had not written a line of Rust for at least a year, my Emacs configuration for this language was broken, and I had lost all my fluency in this language. Still, I was not expecting to pick OCaml when I started this project. , and by the end of the week, I had a first working prototype I called Spatial Sway . It works pretty well; well enough that I am using it daily for several weeks now. It feels clunky at times, but it works well, and I have been able to write a Waybar configuration heavily inspired on Material Shell UI.

Overall, I am pretty satisfied with this turnout. Writing a hobbyist software project is always nice, but the ones you can integrate in your daily workflow are the best one. The last time I did that was keyrd , my little keystrokes counting daemon19,970,965 since I started using it at the time of writing this article .

Anyway, lots remains to be said about Spatial Sway, but I might save it for a bit later. I still have some key features to implement (notably, moving a window to another workspace), then I will probably try to advertise it a bit. I am under the impression this project could be of interest for others, and I would love to see it used by folks willing to give a Material Shell-like experience a try, without having to deal with Gnome Shell. By the way, considering Sway is a drop-in replacement for i3, and that it implements the exact same IPC protocol, there is no reason why Spatial Sway is actually Sway specific, and I will rename it Spatial Shell at some point.

Mandatory screenshot of Spatial Sway.

This Website

On a side note, I have started to refine the layout of this website a bit. Similarly, I have written a new, curated home page where I want to highlight the most recent things I have published on the Internet.

I have been experimenting with Alectryon as a way to replace coqdoc, to improve the readability of my Coq-related articles. Unfortunately, it looks like this tool is missing a key feature I need . I might try to get my hand dirty and implement it myself if I find the time and the motivation in the following weeks.

Finally, reading about how Xe Iaso’s talk about how she generates her blog was very inspiring to me. I can only suggest that you have a look.

Though not to the same extent, I also think I have spent way too much effort in my website. Most of my Coq-related articles are actual Coq program, expect the articles about coqffi which are Org mode literate programs. Hell, this website itself used to be a literate program of the sort, until I stopped using my homegrown literate programming toolchain cleopatra last month. At some point, I have even spent a bit of time to ensure most of the pages of this website were granted a 100/100 on websites like PageSpeed InsightGood news, I’ve just checked, and it still is! . I had almost forgotten.

A lot remains to be done, but watching this talk made me reflect on the job done. And opened my eyes to a new perspective, too. We will see what translates into reality.

What happened in August 2022?

August 15, 2022

What happened in August 2022?

emacs meta

Without further ado, let’s take a look at what was achieved for the last thirty days or so.

Emacs

I have started tweaking and improving my Emacs configuration againAfter having used Emacs for seven years now, I am nowhere close to consider my configuration as a done project. I really envy developers who are using their editor with little to no customization. .

Theme Selection Menu

The change I am the most excited about is that I have finally reduced the boilerplate in need to write to use a new theme. I am very indecisive when it comes to theming. I like to have my choices, and I get tired of any color scheme pretty quickly. As a consequence, I introduced a customizable variable to let me select a theme dynamically, and have this choice persist across Emacs session.

I have a Hydra menu that allows me to select which theme I want to use for the time being. It looks like this.

A Hydra menu for selecting a theme.

But adding new entries to this menu was very cumbersome, and mostly boilerplate that I know a good macro could abstract away. And I can finally report that I was right all along. I have my macros now, and they allow me to have the Hydra menu above generated with these simple lines of code.

(use-theme ancientless "a" :straight nil :load-path "~/.emacs.d/lisp")
(use-theme darkless "d" :straight nil :load-path "~/.emacs.d/lisp")
(use-theme brightless "b" :straight nil :load-path "~/.emacs.d/lisp")
(use-theme monotropic "m")
(use-theme monokai "M")
(use-theme nothing "n")
(use-theme eink "e")
(use-theme dracula "D")
(use-theme chocolate "c")
(use-themes-from tao-theme
                 '(("tl" . tao-yang)
                   ("td" . tao-yin)))

Eldoc and Flycheck Popups

I have been experimenting with several combinations of packages to have Eldoc and Flycheck using pop-up-like mechanisms to report things to me, instead of the echo area.

The winning setup for now is the one that uses the quick-peek package . That is, flycheck-inline (customized to use quick-peek, as suggested in their README), and eldoc-overlay . This works well enough, so the pop-ups of eldoc are maybe a bit too distracting.

#flycheck-inline in action with an OCaml compilation error.

In my quest for pop-ups, I ran into several issues with the packages I tried out. For instance, eldoc-box was very nice, but also very slow for some reason. It turns out there was an issue about that slowness, wherein the culprit was identified. This allowed me to submit a pull request that got merged rather quickly .

Similarly, after a packages update, I discovered flycheck-ocaml was no longer working, and submit a patch to fix the issue .

This Website

I have not been investing a lot of time in this website for the past six years or so. This month, things change a bit on that side too.

New Contents

First, I have published a (short) article on higher-order polymorphism in OCaml. The goal was for me to log somewhere the solution for an odd problem I was confronted to at $WORK, but the resulting article was not doing a great job as conveying this. In particular, two comments on Reddit motivated me to rework it, and I am glad I did. I hope you enjoy the retake.

Once this was out of the way, I decided that generating this website was taking way too much time for no good enough reason. The culprit was cleopatra, a toolchain I had developed in 2020 to integrate the build process of this website as additional contents that I thought might interest people. The sad things were: cleopatra was adding a significant overhead, and I never take the time to actually document them properly.

Under the Hood

Overall, the cost of using cleopatra was not worth the burden, and so I got rid of it. Fortunately, it was not very difficult, since the job of cleopatra was to extracting the generation processes from org files; I just add to implement a small makefile to make use of these files, without having to rely on cleopatra anymore.

This was something I was pondering to do for a long time, and as often in these circumstances, this gave me the extra motivation I needed to tackle other ideas I had in mind for this website. This is why now, rather than starting one Emacs process per Org file I have to process, my build toolchain starts one Emacs server, and later uses emacsclient.

Now, most of the build time is spent by soupault . I guess I will have to spend some time on the Lua plugins I have developed for it at some point.

A New Mailing List

Finally, I have created a public mailing list that is available if you want to start a discussion on one of my articles. Don’t hesitate to use it, or to register to it!

Writing a Function Whose Argument is a Polymorphic Function in OCaml

August 7, 2022

Writing a Function Whose Argument is a Polymorphic Function in OCaml

ocaml

In OCaml, it is not possible to write a function whose argument is a polymorphic function. Trying to write such a function results in the type-checker complaining back at you.

let foo (type a b) id (x : a) (y : b) = (id x, id y)

Line 1, characters 50-51:
1 | let foo (type a b) id (x : a) (y : b) = (id x, id y);;
                                                      ^
Error: This expression has type b but an expression was expected
of type a

When OCaml tries to type check foo, it infers id expects an argument of type a because of id x, then fails when trying to type check id y.

The trick to be able to write foo is to use records. Indeed, while the argument of a function cannot be polymorphic, the field of a record can. This effectively makes it possible to write foo, at the cost of a level of indirection.

type id = {id : 'a. 'a -> 'a}

let foo {id} x y = (id x, id y)

From a runtime perspective, it is possible to tell OCaml to remove the introduced indirection with the unboxed annotation. There is nothing we can do in the source, though. We need to destruct id in foo, and we need to construct it at its call site.

g {id = fun x -> x}

As a consequence, this solution is not a silver bullet, but it is an option that is worth considering if, e.g., it allows us to export a cleaner API to the consumer of a module. Personally, I have been considering this trick recently to remove the need for a library to be implemented as a functor.

How I Use Stacked Git at $WORK

January 16, 2022

How I Use Stacked Git at `$WORK`

git workflow

According to my Lobste.rs history , I have run into Stacked Git in early April 2021, and I remember that its promises hit a soft spot. A few weeks later, I was submitting a pull request to teach Stacked Git to sign commits . It was all I needed to start using it at $WORK, and now it has become a cornerstone of my development workflow.

What is Stacked Git?

Before going any further, it is probably a good idea to take a moment and present Stacked Git. The website introduces the tool as follows:

Stacked Git, StGit for short, is an application for managing Git commits as a stack of patches.

There are a few things to unpack here. First and as its name suggests, Stacked Git is a tool built on top of GitMy main takeaway from my Pijul adventure is connected to this. Git is not limited to the git binary. Git comes with a collection of powerful forges, nice editor plugins, and years of good practices. To this day, it’s neither the bugs nor the breaking changes that made me quite Pijul. Those were expected. What I naively did not anticipate is the dry feeling that Pijul was just the pijul binary, which left me with a lot of tasks to do manually. . It is not a brand new VCS, and as a consequence you keep using all your existing tools and pluginsI am looking at you, Magit. . Secondly, Stacked Git helps you curate your Git history, by turning your commits into patches, and your branches into stacks of patches. This speaks to me, maybe because I have been fascinated by email-based workflows for quite some time.

To me, the two core features of Stacked Git are (1) allowing you to name your commits, and (2) to navigate among them. Together, they create a wonderful companion to help you keep your history clean.

My Subset of Stacked Git

I do not want this article to be a Stacked Git tutorial. Fortunately, I don’t really use the tool at its full potential. I only care about a relatively small subset of commands I feel comfortable with and use daily.

First, to decide which commits are part of my “stack of patches,” I can count of these commands:

stg new NAME creates an empty commit, and gives it the name NAME. Having a way to identify a patch with a meaningful name that is resistant to rebase and amend is very nice. These are two properties commit hashes do not have.
stg uncommit NAME names the most recent commit under my stack with NAME and integrates it into it. I do this when I am tasked to work on a merge request made by a colleague, for instance.
stg commit removes from my stack its last patch. I do this when said commit has been merged into master.

Once my stack of patches is ready, the fun begins.

At a given time, a patch can either be (1) applied, (2) unapplied, or (3) hidden. On the one hand, if a patch is applied it is part of the Git history. On the other hand, unapplying a patch means removing it from the working branch (but not from the stack of patches of Stacked Git). If a patch becomes irrelevant, but you don’t want to remove it entirely because it can become handy later, you can hide it. A hidden patch sits beside the stack of patches, and can be reintegrated if need be.

Analogous to git log ---which allows you to visualize your Git history---, stg series gives you a view of the state of your stack of patches. Patches prefixed with + (or >) are applied, while - means the patch is unapplied.

Then,

stg pop unapplies the patch on top of the list of applied patches.
stg push applies the patch on the bottom of the list of unapplied patches.
stg goto NAME unapplies or applies the necessary patches so that NAME becomes the top patch of the list of applied patches.

Both HEAD and the work tree are updated accordingly.

In addition, stg sink and stg float allow reorganizing your stack of patches, moving patches around. Basically, they are like git rebase -i, but without having to use $EDITOR.

Modifying patches is done with stg refresh. It’s akin to git commit --amend, except it is more powerful because you can modify any applied patches with the -p option. I’d always encourage you to stg goto first, because stg refresh -p remains unfortunately error-prone (nothing prevents you from targeting the wrong patch). But when used carefully, it can be very handy.

Finally, stg rebase REF moves your stack of patches on top of REFStacked Git is supposedly able to detect, during a rebase, which of your patches have been applied to your target branch. I’d rather use stg uncommit before doing the rebase, though. . It is akin to git rebase --onto, but more straightforward. What happens is Stacked Git pop all the patches of my stack, reset the HEAD of the current branch to REF, and tries applying the patches one by one In case of conflicts, the process stop, and I am left with an empty patch, and a dirty work tree with conflicts to solve. The hidden gem is that, contrary to git rebase, the repository is not “in the middle of a rebase.”

Suppose there are many conflicting patches still waiting in my stack of patches, and an urgent task I need to take care of first. I can just leave them here. I can switch to another branch, and when I come back, I get my patches back. I call this feature “incremental rebases.”

And that is basically it. In a nutshell, Stacked Git equips commits with the same features as branches.

My Stacked Git Workflow

As mentioned in the introduction of this article, Stacked Git has become a cornerstone of my workflow. I’ve been asked a few times what this workflow is, and why Magit is not enoughIt’s always about Magit. ;) . So let’s try to do that. But first, a warning. Yes, because Stacked Git is only a wrapper above Git, everything I will explain can be achieved using Git alone, especially if you are a Magit wizard.

Stacked Git makes just everything so more convenient to me.

Planning My Commits Ahead Of Time

I’ve been introduced to Git with a pretty simple workflow: I am supposed to start working on a feature, and once it’s ready, I can commit, and move on to the next task on my to-do list.

To me, this approach is backward. It makes you set your intent after the fact. With Stacked Git, I often try to plan my final history /before writing the very first line of code/. Using stack new, I create my patches, and take the time to write their description. It helps me visualize where I want to go. Then, I use stack goto to go back to the beginning of my stack, and start working.

It is not, and cannot be, an exact science. I often have to refine them as my work progresses. Yet, I think my Git history is cleaner, more focused, since I have started this exercise.

Getting My Fixup Commits Right

Reviews are a fundamental aspect of a software developer job. At $WORK, we use Gitlab and their merge requests workflow, which I find very annoying, because it does not provide meaningful ways to compare two versions of your submissionThere is a notion of “versions” in Gitlab, but its ergonomics fall short of my expectations for such a tool. .

What we end up doing is creating “fixup commits,” and we push them to Gitlab so that reviewers can easily verify that their feedback has correctly been taken into account.

A fixup commit is a commit that will eventually be squashed into another. You can understand it as a delayed git commit --amend. Git has some built-in features to manipulate them. You create them with git commit --fixup=<HASH>, and they are interpreted in a specific manner by git rebase -i. But they have always felt to me like a sordid hack. It is way too easy to create a fixup commit that targets the wrong commit, and you can end up with strange conflicts when you finally squash them. That being said, if used carefully, they are a powerful tool to keep a Git history clean.

I am not sure we are using them carefully, though.

Some reviews can be excruciating, with dozens of comments to address, and theoretically as many fixup commits to create. Then you push all of them on Gitlab, and days later, after the green light from the reviewer, you get to call git rebase and discover your history is broken, you have tones of conflicts to fix, and you’re good for a long afternoon of untangling.

The main reason behind this mess is that you end up fixing a commit from the HEAD of your working branch, not the commit itself. But with Stacked Git, things are different. With stg goto, I put my working tree in the best state possible to fix a commit: the commit itself. I can use stg new to create a fixup commit, with a meaningful name. Then, I am forced to deal with the potential conflicts it brings when I call stg push.

Once my reviewer is happy with my work, I can call stg squash. It is less automated than git rebase -i, but the comfort I gained during the development is worth this little annoyance.

Managing Stacked Merge Requests

At $WORK, we are trying to change how we deliver new features to our master branch. More precisely, we want to merge smaller contributions more frequently. We have had our fair share of large and complex merge requests that were a nightmare to review in the past, and it’s really not a fun position to be put in.

For a few months, I have been involved in a project wherein we decided /not/ to fall in the same trap again. We agreed on a “planning of merge requests” and started working. The first merge request was soon opened. We’ve nominated an “owner” to take care of the review, and the rest of the team carried on. Before the first merge request was merged, the second one was declared ready, and another owner was appointed. Then, the owner of the first merge request had a baby, and yours truly ended up having to manage two interdependent merge requests.

It turns out Stacked Git is a wonderful tool to help me keep this under control.

I only have one branch, and I use the same workflow to deal with feedback, even if they are coming from more than one merge request. To remember the structure of everything, I just prefix the name of my patches with a merge request nickname. So my stack will look something like this:

+ mr1-base
+ mr1-tests
+ mr1-doc
> mr2-command
- mr2-tests

A reviewer leaves a hard-truth comment that requires a significant rework of the oldest merge request? stg goto reverts my work tree in the appropriate state, and stg push allows me to deal with conflicts one patch at a time. If I need to spend more time on the oldest merge request at some point, I can continue my work, knowing the patches related to the newest one are awaiting in my stack.

The most annoying part is when the time comes to push everything. I need to stg goto at the last patch of each merge request, and git push HEAD:the-branch. It’s not horrible. But I will probably try to automate it at some point.

Conclusion

Overall, I am really thankful to Stacked Git’s authors! Thank you! You are making my interactions with Git fun and carefree. You provide me some of the convenience of patch-based VCS like Darcs and Pijul , but without sacrificing the power of Git.

I encourage anyone to at least give it a try, and I really hope I will be able to contribute back to Stacked Git in the near future.

Implementing an Echo Server in Coq with coqffi.1.0.0

December 10, 2020

Implementing an Echo Server in Coq with `coqffi.1.0.0`

coq ocaml coqffi

In this article, we will demonstrate how coqffi can be used to implement an echo server, i.e., a TCP server which sends back any input it receives from its clients. In addition to coqffi, you will need to install coq-simple-io. The latter is available in the released repository of the Opam Coq Archive .

opam install coq-coqffi coq-simple-io

Besides, you can download the source tree presented in this article if you want to try to read the source directly, or modify it to your taste.

Project Layout

Before diving too much into the implementation of our echo server, we first give an overview of the resulting project’s layout. Since we aim at implementing a program, we draw our inspiration from the idiomatic way of organizing a OCaml project.

We have three directories at the root of the project.

ffi/ contains the low-level OCaml code: It provides an OCaml library (ffi), and a Coq theory (FFI) which gathers the FFI modules generated by coqffi.
src/ contains the Coq implementation of our echo server: It provides a Coq theory (Echo) which depends on the FFI theory the SimpleIO theory of coq-simple~io. This theory provides the implementation of our echo server in Coq.
bin/ contains the pieces of code to get an executable program: It contains a Coq module (echo.v) which configures and uses the extraction mechanism to generate an OCaml module (echo.ml). This OCaml module can be compiled to get an executable program.

Note that we could have decided to only have one Coq theory. We could also have added a fourth directory (theories/) for formal verification specific code, but this is out of the scope of this tutorial.

Overall, we use dune to compile and compose the different parts of the echo server. dune has a native —yet unstable at the time of writing— support for building Coq projects, with very convenient stanzas like coq.theory and coq.extraction.

The following graph summarizes the dependencies between each component (plain arrows symbolize software dependencies).

The echo server dependency graph. Dashed boxes are generated.

We enable Coq-related stanza with (using coq 0.2) in the dune-project. file.

(lang dune 2.7)
(using coq 0.2)

The rest of this tutorial proceeds by diving into each directory.

FFI Bindings

Our objective is to implement an echo server, i.e., a server which (1) accepts incoming connections, and (2) sends back any incoming messages. We will consider two classes of effects. One is related to creating and manipulating TCP sockets. The other is dedicated to process management, more precisely to be able to fork when receiving incoming connections.

Therefore, the ffi library will provide two modules. Likewise, the FFI theory will provide two analogous modules generated by coqffi.

In the ffi/ directory, we add the following stanza to the dune file.

(library
  (name ffi)
  (libraries unix))

dune will look for any .ml and .mli files within the directory and will consider they belong to the ffi library. We use the unix library to implement the features we are looking for.

Then, we add the following stanza to the dune file of the ffi/ directory.

(coq.theory
  (name FFI))

This tells dune to look for .v file within the ffi/ directory, in order to build them with Coq. A nice feature of dune is that if we automatically generate Coq files, they will be automatically “attached” to this theory.

Sockets

Sockets are boring. The following OCaml module interface provides the necessary type and functions to manipulate them.

type socket_descr

val open_socket : string -> int -> socket_descr
val listen : socket_descr -> unit
val recv : socket_descr -> string
val send : socket_descr -> string -> int
val accept_connection : socket_descr -> socket_descr
val close_socket : socket_descr -> unit

Our focus is how to write the interface modules for coqffi. Since the object of this tutorial is not the implementation of an echo server in itself, the implementation details of the ffi library will not be discussed, but is provided at the end of this article.

dune generates .cmi files for the .mli files of our library, and provides the necessary bits to easily locate them. Besides, the action stanza can be used here to tell to dune how to generate the module Socket.v from file.cmi. We add the following entry to ffi/dune.

(rule
  (target Socket.v)
  (action (run coqffi %{cmi:socket} -o %{target})))

We call coqffi without any feature-related command-line argument, which means only the simple-io feature is enabled. As a consequence, the socket_descr type is axiomatized in Coq, and in addition to a MonadSocket monad, coqffi will generate an instance for this monad for the IO monad of coq-simple-io.

The stanza generates the following Coq module.

(* This file has been generated by coqffi. *)

Set Implicit Arguments.
Unset Strict Implicit.
Set Contextual Implicit.
Generalizable All Variables.
Close Scope nat_scope.

From CoqFFI Require Export Extraction.
From SimpleIO Require Import IO_Monad.

Axiom socket_descr : Type.

Extract Constant socket_descr => "Ffi.Socket.socket_descr".

(** * Impure Primitives *)

(** ** Monad Definition *)

Class MonadSocket (m : Type -> Type) : Type :=
  { open_socket : string -> i63 -> m socket_descr
  ; listen : socket_descr -> m unit
  ; recv : socket_descr -> m string
  ; send : socket_descr -> string -> m i63
  ; accept_connection : socket_descr -> m socket_descr
  ; close_socket : socket_descr -> m unit
  }.

(** ** [IO] Instance *)

Axiom io_open_socket : string -> i63 -> IO socket_descr.
Axiom io_listen : socket_descr -> IO unit.
Axiom io_recv : socket_descr -> IO string.
Axiom io_send : socket_descr -> string -> IO i63.
Axiom io_accept_connection : socket_descr -> IO socket_descr.
Axiom io_close_socket : socket_descr -> IO unit.

Extract Constant io_open_socket
  => "(fun x1 x2 k__ -> k__ ((Ffi.Socket.open_socket x1 x2)))".
Extract Constant io_listen => "(fun x1 k__ -> k__ ((Ffi.Socket.listen x1)))".
Extract Constant io_recv => "(fun x1 k__ -> k__ ((Ffi.Socket.recv x1)))".
Extract Constant io_send
  => "(fun x1 x2 k__ -> k__ ((Ffi.Socket.send x1 x2)))".
Extract Constant io_accept_connection
  => "(fun x1 k__ -> k__ ((Ffi.Socket.accept_connection x1)))".
Extract Constant io_close_socket
  => "(fun x1 k__ -> k__ ((Ffi.Socket.close_socket x1)))".

Instance IO_MonadSocket : MonadSocket IO :=
  { open_socket := io_open_socket
  ; listen := io_listen
  ; recv := io_recv
  ; send := io_send
  ; accept_connection := io_accept_connection
  ; close_socket := io_close_socket
  }.

(* The generated file ends here. *)

Process Management

In order to avoid a client to block the server by connecting to it without sending anything, we can fork a new process for each client.

type identity = Parent of int | Child

val fork : unit -> identity

This time, the proc.mli module interface introduces a transparent type, /i.e./, it also provides its definition. This is a good use case for the transparent-types feature of coqffi. In the stanza for generating Proc.v, we enable it with the -ftransparent-types command-line argument, like this.

(rule
  (target Proc.v)
  (action (run coqffi -ftransparent-types %{cmi:proc} -o %{target})))

which generates the following Coq module.

(* This file has been generated by coqffi. *)

Set Implicit Arguments.
Unset Strict Implicit.
Set Contextual Implicit.
Generalizable All Variables.
Close Scope nat_scope.

From CoqFFI Require Export Extraction.
From SimpleIO Require Import IO_Monad.

Inductive identity : Type :=
| Parent (x0 : i63) : identity
| Child : identity.

Extract Inductive identity => "Ffi.Proc.identity"
  [ "Ffi.Proc.Parent" "Ffi.Proc.Child" ].

(** * Impure Primitives *)

(** ** Monad Definition *)

Class MonadProc (m : Type -> Type) : Type := { fork : unit -> m identity
                                             }.

(** ** [IO] Instance *)

Axiom io_fork : unit -> IO identity.

Extract Constant io_fork => "(fun x1 k__ -> k__ ((Ffi.Proc.fork x1)))".

Instance IO_MonadProc : MonadProc IO := { fork := io_fork
                                        }.

(* The generated file ends here. *)

We now have everything we need to implement an echo server in Coq.

Implementing an Echo Server

Our implementation will be part of a dedicated Coq theory, called Echo. This is done easily a dune file in the src/ directory, with the following content.

(coq.theory
  (name Echo)
  (theories FFI))

In the rest of this section, we will discuss the content of the unique module of this theory. Hopefully, readers familiar with programming impurity by means of monads will not find anything particularly surprising here.

Let us start with the inevitable sequence of import commands. We use the Monad and MonadFix typeclasses of ExtLib, and our FFI modules from the FFI theory we have previously defined.

From ExtLib Require Import Monad MonadFix.
From FFI Require Import Proc Socket.

Letting Coq guess the type of unintroduced variables using the ` annotation (e.g., in presence of `{Monad m}, Coq understands m is of type Type -> Type) is always nice, so we enable it.

Generalizable All Variables.

We enable the monad notation provided by ExtLib. In this article, we prefer the let* notation (as recently introduced by OCaml) over the <- notation of Haskell, but both are available.

Import MonadLetNotation.
Open Scope monad_scope.

Then, we define a notation to be able to define local, monadic recursive functions using the mfix combinator of the MonadFix typeclass.

Notation "'let_rec*' f x ':`' p 'in' q" :`
  (let f :` mfix (fun f x `> p) in q)
    (at level 61, x pattern, f name, q at next level, right associativity).

Note that mfix does /not/ check whether or not the defined function will terminate (contrary to the fix keyword of Coq). This is fortunate because in our case, we do not want our echo server to converge, but rather to accept an infinite number of connections.

We can demonstrate how this notation can be leveraged by defining a generic TCP server, parameterized by a handler to deal with incoming connections.

Definition tcp_srv `{Monad m, MonadFix m, MonadProc m, MonadSocket m}
    (handler : socket_descr -> m unit)
  : m unit :=
  let* srv := open_socket "127.0.0.1" 8888 in
  listen srv;;

  let_rec* tcp_aux _ :=
    let* client := accept_connection srv in
    let* res := fork tt in
    match res with
    | Parent _ => close_socket client >>= tcp_aux
    | Child =>  handler client
    end
  in

  tcp_aux tt.

The handler for the echo server is straightforward: it just reads incoming bytes from the socket, sends it back, and closes the socket.

Definition echo_handler `{Monad m, MonadSocket m} (sock : socket_descr)
  : m unit :=
  let* msg := recv sock in
  send sock msg;;
  close_socket sock.

Composing our generic TCP server with our echo handler gives us an echo server.

Definition echo_server `{Monad m, MonadFix m, MonadProc m, MonadSocket m}
  : m unit :=
  tcp_srv echo_handler.

Because coqffi has generated typeclasses for the impure primitives of proc.mli and socket.mli, echo_server is polymorphic, and can be instantiated for different monads. When it comes to extracting our program, we will generally prefer the IO monad of coq-simple-io. But we could also imagine verifying the client handler with FreeSpec, or the generic TCP server with Interaction Trees (which support diverging computations). Overall, we can have different verification strategies for different parts of our program, by leveraging the most relevant framework for each part, yet being able to extract it in an efficient form.

The next section shows how this last part is achieved using, once again, a convenient stanza of dune.

Extracting and Building an Executable

The 0.2 version of the Coq-related stanzas of dune provides the coq.extraction stanza, which can be used to build a Coq module expected to generate ml files.

In our case, we will write bin/echo.v to extract the echo_server in a echo.ml module, and uses the executable stanza of dune to get an executable from this file. To achieve this, the bin/dune file simply requires these two stanzas.

(coq.extraction
  (prelude echo)
  (theories Echo)
  (extracted_modules echo))

(executable
  (name echo)
  (libraries ffi))

We are almost done. We now need to write the echo.v module, which mostly consists of (1) providing a MonadFix instance for the IO monad, (2) using the IO.unsafe_run function to escape the IO monad, (3) calling the Extraction command to wrap it up.

From Coq Require Extraction.
From ExtLib Require Import MonadFix.
From SimpleIO Require Import SimpleIO.
From Echo Require Import Server.

Instance MonadFix_IO : MonadFix IO :=
  { mfix := @IO.fix_io }.

Definition main : io_unit :=
  IO.unsafe_run echo_server.

Extraction "echo.ml" main.

Since we are using the i63 type (signed 63bits integers) of the CoqFFI theory, and since i63 is implemented under the hood with Coq primitive integers, we also need to provide a Uint63 module with a of_int function. Fortunately, this module is straightforward to write.

let of_int x = x

And voilà. A call to dune at the root of the repository will build everything (Coq and OCaml alike). Starting the echo server is as simple as

dune exec bin/echo.exe

And connecting to it can be achieved with a program like telnet.

$ telnet 127.0.0.1 8888
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
hello, echo server!
hello, echo server!
Connection closed by foreign host.

Appendix

The `Socket` OCaml Module

There is not much to say, except that (as already stated) we use the Unix module to manipulate sockets, and we attach to each socket a buffer to store incoming bytes.

let buffer_size = 1024

type socket_descr = {
  fd : Unix.file_descr;
  recv_buffer : bytes;
}

let from_fd fd =
  let rbuff = Bytes.create buffer_size in
  { fd ` fd; recv_buffer ` rbuff }

let open_socket hostname port =
  let open Unix in
  let addr = inet_addr_of_string hostname in
  let fd = socket PF_INET SOCK_STREAM 0 in
  setsockopt fd SO_REUSEADDR true;
  bind fd (ADDR_INET (addr, port));
  from_fd fd

let listen sock = Unix.listen sock.fd 1

let recv sock =
  let s = Unix.read sock.fd sock.recv_buffer 0 buffer_size in
  Bytes.sub_string sock.recv_buffer 0 s

let send sock msg =
  Unix.write_substring sock.fd msg 0 (String.length msg)

let accept_connection sock =
  Unix.accept sock.fd |> fst |> from_fd

let close_socket sock = Unix.close sock.fd

The `Proc` OCaml Module

Thanks to the Unix module, the implementation is pretty straightforward.

type identity = Parent of int | Child

let fork x =
  match Unix.fork x with
  | 0 -> Child
  | x -> Parent x

coqffi.1.0.0 In A Nutshell

December 10, 2020

`coqffi.1.0.0` In A Nutshell

coq ocaml coqffi

For each entry of a cmi file (a compiled mli file), coqffi tries to generate an equivalent (from the extraction mechanism perspective) Coq definition. In this article, we walk through how coqffi works.

Note that we do not dive into the vernacular commands coqffi generates. They are of no concern for users of coqffi.

Getting Started

Requirements

The latest version of coqffi (1.0.0~beta8) is compatible with OCaml 4.08 up to 4.14, and Coq 8.12 up top 8.13. If you want to use coqffi, but have incompatible requirements of your own, feel free to submit an issue .

Installing `coqffi`

The recommended way to install coqffi is through the Opam Coq Archive , in the released repository. If you haven’t activated this repository yet, you can use the following bash command.

opam repo add coq-released https://coq.inria.fr/opam/released

Then, installing coqffi is as simple as

opam install coq-coqffi

You can also get the source from the upstream git repository . The README provides the necessary pieces of information to build it from source.

Additional Dependencies

One major difference between Coq and OCaml is that the former is pure, while the latter is not. Impurity can be modeled in pure languages, and Coq does not lack of frameworks in this respect. coqffi currently supports two of them: coq-simple-io and FreeSpec . It is also possible to use it with Interaction Trees , albeit in a less direct manner.

Primitive Types

coqffi supports a set of primitive types, i.e., a set of OCaml types for which it knows an equivalent type in Coq. The list is the following (the Coq types are fully qualified in the table, but not in the generated Coq module as the necessary Import statements are generated too).

OCaml type	Coq type
`bool`	`Coq.Init.Datatypes.bool`
`char`	`Coq.Strings.Ascii.ascii`
`int`	`CoqFFI.Data.Int.i63`
`'a list`	`Coq.Init.Datatypes.list a`
`'a Seq.t`	`CoqFFI.Data.Seq.t`
`'a option`	`Coq.Init.Datatypes.option a`
`('a, 'e) result`	`Coq.Init.Datatypes.sum`
`string`	`Coq.Strings.String.string`
`unit`	`Coq.Init.Datatypes.unit`
`exn`	`CoqFFI.Exn`

The i63 type is introduced by the CoqFFI theory to provide signed primitive integers to Coq users. They are implemented on top of the (unsigned) Coq native integers introduced in Coq 8.13. The i63 type will be deprecated once the support for signed primitive integers is implementedThis is actually one of the sources of incompatibility of coqffi with most recent versions of Coq. .

When processing the entries of a given interface model, coqffi will check that they only use these types, or types introduced by the interface module itself.

Sometimes, you may encounter a situation where you have two interface modules b.mli and b.mli, such that b.mli uses a type introduced in a.mli. To deal with this scenario, you can use the --witness flag to generate A.v. This will tell coqffi to also generate A.ffi; this file can then be used when generating B.v thanks to the -I option. Furthermore, for B.v to compile the --require option needs to be used to ensure the A Coq library (A.v) is required.

To give a more concrete example, given ~a.mli~

type t

and b.mli

type a = A.t

To generate A.v, we can use the following commands:

ocamlc a.mli
coqffi --witness -o A.v a.cmi

Which would generate the following axiom for t.

Axiom t : Type.

Then, generating B.v can be achieved as follows:

ocamlc b.mli
coqffi -I A.ffi -ftransparent-types -r A -o B.v b.cmi

which results in the following output for v:

Require A.

Definition u : Type := A.t.

Code Generation

coqffi distinguishes five types of entries: types, pure values, impure primitives, asynchronous primitives, exceptions, and modules. We now discuss how each one of them is handled.

Types

By default, coqffi generates axiomatized definitions for each type defined in a .cmi file. This means that type t becomes Axiom t : Type. Polymorphism is supported, i.e., type 'a t becomes Axiom t : forall (a : Type), Type.

It is possible to provide a “model” for a type using the coq_model annotation, for instance, for reasoning purposes. That is, we can specify that a type is equivalent to a list.

type 'a t [@@coq_model "list"]

This generates the following Coq definition.

Definition t : forall (a : Type), Type := list.

It is important to be careful when using the =coq_model= annotation. More precisely, the fact that t is a list in the “Coq universe” shall not be used while the implementation phase, only the verification phase.

Unnamed polymorphic type parameters are also supported. In presence of such parameters, coqffi will find it a name that is not already used. For instance,

type (_, 'a) ast

becomes

Axiom ast : forall (b : Type) (a : Type), Type.

Finally, coqffi has got an experimental feature called transparent-types (enabled by using the -ftransparent-types command-line argument). If the type definition is given in the module interface, then coqffi tries to generate an equivalent definition in Coq. For instance,

type 'a llist =
  | LCons of 'a * (unit -> 'a llist)
  | LNil

becomes

Inductive llist (a : Type) : Type :=
| LCons (x0 : a) (x1 : unit -> llist a) : llist a
| LNil : llist a.

Mutually recursive types are supported, so

type even = Zero | ESucc of odd
and odd = OSucc of even

becomes

Inductive odd : Type :=
| OSucc (x0 : even) : odd
with even : Type :=
| Zero : even
| ESucc (x0 : odd) : even.

Besides, coqffi supports alias types, as suggested in this write-up when we discuss witness files.

The transparent-types feature is experimental, and is currently limited to variant types. It notably does not support records. Besides, it may generate incorrect Coq types, because it does not check whether or not the positivity condition is satisfied.

Pure values

coqffi decides whether or not a given OCaml value is pure or impure with the following heuristics:

Constants are pure
Functions are impure by default
Functions with a coq_model annotation are pure
Functions marked with the pure annotation are pure
If the pure-module feature is enabled (-fpure-module), then synchronous functions (which do not live inside the ~Lwt~ monad) are pure

Similarly to types, coqffi generates axioms (or definitions if the coq_model annotation is used) for pure values. Then,

val unpack : string -> (char * string) option [@@pure]

becomes

Axiom unpack : string -> option (ascii * string).

Polymorphic values are supported.

val map : ('a -> 'b) -> 'a list -> 'b list [@@pure]

becomes

Axiom map : forall (a : Type) (b : Type), (a -> b) -> list a -> list b.

Again, unnamed polymorphic typse are supported, so

val ast_to_string : _ ast -> string [@@pure]

becomes

Axiom ast_to_string : forall (a : Type), string.

Impure Primitives

coqffi reserves a special treatment for /impure/ OCaml functions. Impurity is usually handled in pure programming languages by means of monads, and coqffi is no exception to the rule.

Given the set of impure primitives declared in an interface module, coqffi will (1) generate a typeclass which gathers these primitives, and (2) generate instances of this typeclass for supported backends.

We illustrate the rest of this section with the following impure primitives.

val echo : string -> unit
val scan : unit -> string

where echo allows writing something the standard output, and scan to read the standard input.

Assuming the processed module interface is named console.mli, the following Coq typeclass is generated.

Class MonadConsole (m : Type -> Type) := { echo : string -> m unit
                                         ; scan : unit -> m string
                                         }.

Using this typeclass and with the additional support of an additional Monad typeclass, we can specify impure computations which interacts with the console. For instance, with the support of ExtLib, one can write.

Definition pipe `{Monad m, MonadConsole m} : m unit :=
  let* msg := scan () in
  echo msg.

There is no canonical way to model impurity in Coq, but over the years several frameworks have been released to tackle this challenge.

coqffi provides three features related to impure primitives.

`simple-io`

When this feature is enabled, coqffi generates an instance of the typeclass for the =IO= monad introduced in the coq-simple-io package

Axiom io_echo : string -> IO unit.
Axiom io_scan : unit -> IO string.

Instance IO_MonadConsole : MonadConsole IO := { echo := io_echo
                                              ; scan := io_scan
                                              }.

It is enabled by default, but can be disabled using the -fno-simple-io command-line argument.

`interface`

When this feature is enabled, coqffi generates an inductive type which describes the set of primitives available, to be used with frameworks like FreeSpec or Interactions Trees .

Inductive CONSOLE : Type -> Type :=
| Echo : string -> CONSOLE unit
| Scan : unit -> CONSOLE string.

Definition inj_echo `{Inject CONSOLE m} (x0 : string) : m unit :=
  inject (Echo x0).

Definition inj_scan `{Inject CONSOLE m} (x0 : unit) : m string :=
  inject (Scan x0).

Instance Inject_MonadConsole `{Inject CONSOLE m} : MonadConsole m :=
  { echo := inj_echo
  ; scan := inj_scan
  }.

Providing an instance of the form forall i, Inject i M is enough for your monad M to be compatible with this featureSee for instance how FreeSpec implements it ). .

`freespec`

When this feature in enabled, coqffi generates a semantics for the inductive type generated by the interface feature.

Axiom unsafe_echo : string -> unit.
Axiom unsafe_scan : uint -> string.

Definition console_unsafe_semantics : semantics CONSOLE :=
  bootstrap (fun a e =>
    local match e in CONSOLE a return a with
          | Echo x0 => unsafe_echo x0
          | Scan x0 => unsafe_scan x0
          end).

Asynchronous Primitives

coqffi also reserves a special treatment for asynchronous primitives —i.e., functions which live inside the Lwt monad— when the lwt feature is enabled.

The treatment is very analogous to the one for impure primitives: (1) a typeclass is generated (with the _Async suffix), and (2) an instance for the Lwt monad is generated. Besides, an instance for the “synchronous” primitives is also generated for Lwt. If the interface feature is enabled, an interface datatype is generated, which means you can potentially use Coq to reason about your asynchronous programs (using FreeSpec and alike, although the interleaving of asynchronous programs in not yet supported in FreeSpec).

By default, the type of the Lwt monad is Lwt.t. You can override this setting using the --lwt-alias option. This can be useful when you are using an alias type in place of Lwt.t.

Exceptions

OCaml features an exception mechanism. Developers can define their own exceptions using the exception keyword, whose syntax is similar to the constructors’ definition. For instance,

exception Foo of int * bool

introduces a new exception Foo which takes two parameters of type int and bool. Foo (x, y) constructs of value of type exn.

For each new exception introduced in an OCaml module, coqffi generates (1) a so-called “proxy type,” and (2) conversion functions to and from this type.

Coming back to our example, the “proxy type” generates by coqffi is

Inductive FooExn : Type :=
| MakeFooExn (x0 : i63) (x1 : bool) : FooExn.

Then, coqffi generates conversion functions.

Axiom exn_of_foo : FooExn -> exn.
Axiom foo_of_exn : exn -> option FooExn.

Besides, coqffi also generates an instance for the Exn typeclass provided by the CoqFFI theory:

Instance FooExn_Exn : Exn FooExn :=
  { to_exn := exn_of_foo
  ; of_exn := foo_of_exn
  }.

Under the hood, exn is an extensible datatype , and how coqffi supports it will probably be generalized in future releases.

Finally, coqffi has a minimal support for functions which may raise exceptions. Since OCaml type system does not allow to identify such functions, they need to be annotated explicitly, using the =may_raise= annotation. In such a case, coqffi will change the return type of the function to use the =sum= Coq inductive type.

For instance,

val from_option : 'a option -> 'a [@@may_raise] [@@pure]

becomes

Axiom from_option : forall (a : Type), option a -> sum a exn.

Modules

Lastly, coqffi supports OCaml modules described within mli files, when they are specified as module T : sig ... end. For instance,

module T : sig
  type t

  val to_string : t -> string [@@pure]
end

becomes

Module T.
  Axiom t : Type.

  Axiom to_string : t -> string.
End T.

As of now, the following construction is unfortunately not supported, and will be ignored by coqffi:

module S = sig
  type t

  val to_string : t -> string [@@pure]
end

module T : S

Moving Forward

coqffi comes with a comprehensive man page. In addition, the interested reader can proceed to the next article of this series, which explains how coqffi can be used to easily implement an echo server in Coq.

Pattern Matching on Types and Contexts

August 28, 2020

Pattern Matching on Types and Contexts

coq

In the a previous article of our series on Ltac, we have shown how tactics allow for constructing Coq terms incrementally. Ltac programs (“proof scripts”) generate terms, and the shape of said terms can be very different regarding the initial context. For instance, induction x will refine the current goal by using an inductive principle dedicated to the type of x.

This is possible because Ltac allows for pattern matching on types and contexts. In this article, we give a short introduction on this feature of key importance.

To `lazymatch` or to `match`

Gallina provides one pattern matching construction, whose semantics is always the same: for a given term, the first pattern to match will always be selected. On the contrary, Ltac provides several pattern matching constructions with different semantics. This key difference has probably been motivated because Ltac is not a total language: a tactic may not always succeed.

Ltac programmers can use match or lazymatch. One the one hand, with match, if one pattern matches, but the branch body fails, Coq will backtrack and try the next branch. On the other hand, lazymatch will stop with an error.

So, for instance, the two following tactics will print two different messages:

Ltac match_failure :=
  match goal with
  | _
    => fail "fail in first branch"
  | _
    => fail "fail in second branch"
  end.

Ltac lazymatch_failure :=
  lazymatch goal with
  | _
    => fail "fail in first branch"
  | _
    => fail "fail in second branch"
  end.

We can try that quite easily by starting a dumb goal (eg. Goal (True).) and call our tactic. For match_failure, we get:

Ltac call to "match_failure" failed.
Error: Tactic failure: fail in second branch.

On the other hand, for lazymatch_failure, we get:

Ltac call to "match_failure'" failed.
Error: Tactic failure: fail in first branch.

Moreover, pattern matching allows for matching over patterns, not just constants. Here, Ltac syntax differs from Gallina’s. In Gallina, if a variable name is used in a pattern, Gallina creates a new variable in the scope of the related branch. If a variable with the same name already existed, it is shadowed. On the contrary, in Ltac, using a variable name as-is in a pattern implies an equality check.

To illustrate this difference, we can take the example of the following tactic.

Ltac successive x y :=
  lazymatch y with
  | S x => idtac
  | _ => fail
  end.

successive x y will fail if y is not the successor of x. On the contrary, the “syntactically equivalent” function in Gallina will exhibit a totally different behavior.

Definition successor (x y : nat) : bool :=
  match y with
  | S x => true
  | _ => false
  end.

Here, the first branch of the pattern match is systematically selected when y is not O, and in this branch, the argument x is shadowed by the predecessor of y.

For Ltac to adopt a behavior similar to Gallina, the ? prefix shall be used. For instance, the following tactic will check whether its argument has a known successor, prints it if it does, or fail otherwise.

Ltac print_pred_or_zero x :=
  match x with
  | S ?x => idtac x
  | _ => fail
  end.

On the one hand, print_pred_or_zero 3 will print 2. On the other hand, if there exists a variable x : nat in the context, calling print_pred_or_zero x will fail, since the exact value of x is not known.

Pattern Matching on Types with `type of`

The type of operator can be used in conjunction to pattern matching to generate different terms according to the type of a variable. We could leverage this to reimplement induction for instance.

As an example, we propose the following not_nat tactic which, given an argument x, fails if x is of type nat.

Ltac not_nat x :=
  lazymatch type of x with
  | nat => fail "argument is of type nat"
  | _ => idtac
  end.

With this definition, not_nat true succeeds since true is of type bool, and not_nat O since O encodes $0$ in nat.

We can also use the ? prefix to write true pattern. For instance, the following tactic will fail if the type of its supplied argument has at least one parameter.

Ltac not_param_type x :=
  lazymatch type of x with
  | ?f ?a => fail "argument is of type with parameter"
  | _ => idtac
  end.

Both not_param_type (@nil nat) of type list nat and @eq_refl nat 0 of type 0 = 0 fail, but not_param_type 0 of type nat succeeds. *)

Pattern Matching on the Context with `goal`

Lastly, we discuss how Ltac allows for inspecting the context (i.e., the hypotheses and the goal) using the goal keyword.

For instance, we propose a naive implementation of the subst tactic as follows.

Ltac subst' :=
  repeat
    match goal with
    | H :  ?x = _ |- context[?x]
      => repeat rewrite H; clear H
    end.

With goal, patterns are of the form H : (pattern), ... |- (pattern).

At the left side of |-, we match on hypotheses. Beware that contrary to variable names in pattern, hypothesis names behaves as in Gallina (i.e., fresh binding, shadowing, etc.). In the branch, we are looking for equations, i.e., a hypothesis of the form ?x = _.
At the right side of |-, we match on the goal.

In both cases, Ltac makes available an interesting operator, context[(pattern)], which is satisfied if (pattern) appears somewhere in the object we are pattern matching against. So, the branch of the match reads as follows: we are looking for an equation H which specifies the value of an object x which appears in the goal. If such an equation exists, subst' tries to rewrite x as many times as possible.

This implementation of subst' is very fragile, and will not work if the equation is of the form _ = ?x, and it may behave poorly if we have “transitive equations”, such as there exists hypotheses ?x = ?y and ?y = _. Motivated readers may be interested in proposing a more robust implementation of subst'.

Ltac is an Imperative Metaprogramming Language

August 28, 2020

Ltac is an Imperative Metaprogramming Language

coq

Coq is often depicted as an interactive proof assistant, thanks to its proof environment. Inside the proof environment, Coq presents the user a goal, and said user solves said goal by means of tactics which describes a logical reasoning. For instance, to reason by induction, one can use the induction tactic, while a simple case analysis can rely on the destruct or case_eq tactics, etc. It is not uncommon for new Coq users to be introduced to Ltac, the Coq default tactic language, using this proof-centric approach. This is not surprising, since writing proofs remains the main use case for Ltac. In practice, though, this discourse remains an abstraction which hides away what actually happens under the hood when Coq executes a proof script.

To really understand what Ltac is about, we need to recall ourselves that Coq kernel is mostly a type checker. A theorem statement is expressed as a “type” (which lives in a dedicated sort called Prop), and a proof of this theorem is a term of this type, just like the term S (S O) ( $2$ ) is of type nat. Proving a theorem in Coq requires to construct a term of the type encoding said theorem, and Ltac allows for incrementally constructing this term, one step at a time.

Ltac generates terms, therefore it is a metaprogramming language. It does it incrementally, by using primitives to modifying an implicit state, therefore it is an imperative language. Henceforth, it is an imperative metaprogramming language.

To summarize, a goal presented by Coq inside the environment proof is a hole within the term being constructed. It is presented to users as:

A list of “hypotheses,” which are nothing more than the variables in the scope of the hole
A type, which is the type of the term to construct to fill the hole

We illustrate what happens under the hood of Coq executes a simple proof script. One can use the Show Proof vernacular command to exhibit this.

We illustrate how Ltac works with the following example.

Theorem add_n_O : forall (n : nat), n + O = n.
Proof.

The Proof vernacular command starts the proof environment. Since no tactic has been used, the term we are trying to construct consists solely in a hole, while Coq presents us a goal with no hypothesis, and with the exact type of our theorem, that is forall (n : nat), n + O = n.

A typical Coq course will explain to students the intro tactic allows for turning the premise of an implication into an hypothesis within the context.

C \vdash P \rightarrow Q

becomes

C,\ P \vdash Q

This is a fundamental rule of deductive reasoning, and intro encodes it. It achieves this by refining the current hole into an anonymous function. When we use

  intro n.

it refines the term

  ?Goal1

into

  fun (n : nat) => ?Goal2

The next step of this proof is to reason about induction on n. For nat, it means that to be able to prove

\forall n, \mathrm{P}\ n

we need to prove that

$\mathrm{P}\ 0$
$\forall n, \mathrm{P}\ n \rightarrow \mathrm{P}\ (S n)$

The induction tactic effectively turns our goal into two subgoals. But why is that? Because, under the hood, Ltac is refining the current goal using the nat_ind function automatically generated by Coq when nat was defined. The type of nat_ind is

  forall (P : nat -> Prop),
    P 0
    -> (forall n : nat, P n -> P (S n))
    -> forall n : nat, P n

Interestingly enough, nat_ind is not an axiom. It is a regular Coq function, whose definition is

  fun (P : nat -> Prop) (f : P 0)
      (f0 : forall n : nat, P n -> P (S n)) =>
  fix F (n : nat) : P n :=
    match n as n0 return (P n0) with
    | 0 => f
    | S n0 => f0 n0 (F n0)
    end

So, after executing

  induction n.

The hidden term we are constructing becomes

  (fun n : nat =>
     nat_ind
       (fun n0 : nat => n0 + 0 = n0)
       ?Goal3
       (fun (n0 : nat) (IHn : n0 + 0 = n0) => ?Goal4)
       n)

And Coq presents us two goals.

First, we need to prove $\mathrm{P}\ 0$ , i.e., $0 + 0 = 0$ . Similarly to Coq presenting a goal when what we are actually doing is constructing a term, the use of $=$ and $+$ (i.e., the Coq notations mechanism) hides much here. We can ask Coq to be more explicit by using the vernacular command Set Printing All to learn that when Coq presents us a goal of the form 0 + 0 = 0, it is actually looking for a term of type @eq nat (Nat.add O O) O.

Nat.add is a regular Coq (recursive) function.

  fix add (n m : nat) {struct n} : nat :=
    match n with
    | 0 => m
    | S p => S (add p m)
    end

Similarly, eq is not an axiom. It is a regular inductive type, defined as follows.

Inductive eq (A : Type) (x : A) : A -> Prop :=
| eq_refl : eq A x x

Coming back to our current goal, proving @eq nat (Nat.add 0 0) 0That is, 0 + 0 = 0 requires to construct a term of a type whose only constructor is eq_refl. eq_refl accepts one argument, and encodes the proof that said argument is equal to itself. In practice, Coq type checker will accept the term @eq_refl _ x when it expects a term of type @eq _ x y if it can reduce x and y to the same term.

Is it the case for @eq nat (Nat.add 0 0) 0? It is, since by definition of Nat.add, Nat.add 0 x is reduced to x. We can use the reflexivity tactic to tell Coq to fill the current hole with eq_refl.

  + reflexivity.

Suspicious readers may rely on Show Proof to verify this assertion.

  (fun n : nat =>
     nat_ind
       (fun n0 : nat => n0 + 0 = n0)
       eq_refl
       (fun (n0 : nat) (IHn : n0 + 0 = n0) => ?Goal4)
       n)

?Goal3 has indeed be replaced by eq_refl.

One goal remains, as we need to prove that if n + 0 = n, then S n + 0 = S n. Coq can reduce S n + 0 to S (n + 0) by definition of Nat.add, but it cannot reduce S n more than it already is.

  + cbn.

We cannot just use reflexivity here (i.e., fill the hole with eq_refl), since S (n + 0) and S n cannot be reduced to the same term.

However, at this point of the proof, we have the IHn hypothesis (i.e., the IHn argument of the anonymous function whose body we are trying to construct). The rewrite tactic allows for substituting in a type an occurrence of x by y as long as we have a proof of x = y. *)

    rewrite IHn.

Similarly to induction using a dedicated function, rewrite refines the current hole with the eq_ind_r functionAgain, not an axiom. . Replacing n + 0 with n transforms the goal into S n = S n. Here, we can use reflexivity (i.e., eq_refl) to conclude. *)

    reflexivity.

After this last tactic, the work is done. There is no more goal to fill inside the proof term that we have carefully constructed.

  (fun n : nat =>
     nat_ind
       (fun n0 : nat => n0 + 0 = n0)
       eq_refl
       (fun (n0 : nat) (IHn : n0 + 0 = n0) =>
          eq_ind_r (fun n1 : nat => S n1 = S n0) eq_refl IHn)
       n)

We can finally use Qed or Defined to tell Coq to type check this term. That is, Coq does not trust Ltac, but rather type check the term to verify it is correct. This way, in case Ltac has a bug which makes it construct an ill-formed type, at the very least Coq can reject it.

Qed.

In conclusion, tactics are used to incrementally refine hole inside a term until the latter is complete. They do it in a very specific manner, to encode certain reasoning rules.

On the other hand, the refine tactic provides a generic, low-level way to do the same thing. Knowing how a given tactic works allows for mimicking its behavior using the refine tactic.

If we take the previous theorem as an example, we can prove it using this alternative proof script.

Theorem add_n_O' : forall (n : nat), n + O = n.
Proof.
  refine (fun n => _).
  refine (nat_ind (fun n => n + 0 = n) _ _ n).
  + refine eq_refl.
  + refine (fun m IHm => _).
    refine (eq_ind_r (fun n => S n = S m) _ IHm).
    refine eq_refl.
Qed.

Mixing Ltac and Gallina for Fun and Profit

July 26, 2020

Mixing Ltac and Gallina for Fun and Profit

coq

One of the most misleading introductions to Coq is to say that “Gallina is for programs, while tactics are for proofs.” Indeed, in Coq we construct terms of given types, always. Terms encodes both programs and proofs about these programs. Gallina is the preferred way to construct programs, and tactics are the preferred way to construct proofs.

The key word here is “preferred.” We do not always need to use tactics to construct a proof term. Conversely, there are some occasions where constructing a program with tactics become handy. Furthermore, Coq actually allows for mixing together Ltac and Gallina.

In the previous article of this series, we discuss how Ltac provides two very interesting features:

With match goal with it can inspect its context
With match type of _ with it can pattern matches on types

It turns out these features are more than handy when it comes to metaprogramming (that is, the generation of programs by programs).

A Tale of Two Worlds, and Some Bridges

Constructing terms proofs directly in Gallina often happens when one is writing dependently typed definition. For instance, we can write a type-safe from_option function (inspired by this very nice write-up ) such that the option to unwrap shall be accompanied by a proof that said option contains something. This extra argument is used in the None case to derive a proof of False, from which we can derive anything.

Definition is_some {α} (x : option α) : bool :=
  match x with Some _ => true | None => false end.

Lemma is_some_None {α} (x : option α)
  : x = None -> is_some x <> true.
Proof. intros H. rewrite H. discriminate. Qed.

Definition from_option {α}
    (x : option α) (some : is_some x = true)
  : α :=
  match x as y return x = y -> α with
  | Some x => fun _ => x
  | None => fun equ => False_rect α (is_some_None x equ some)
  end eq_refl.

In from_option, we construct two proofs without using tactics:

False_rect α (is_some_None x equ some) to exclude the absurd case
eq_refl in conjunction with a dependent pattern matching (if you are not familiar with this trick: the main idea is to allow Coq to “remember” that x = None in the second branch)

We can use another approach. We can decide to implement from_option with a proof script.

Definition from_option' {α}
    (x : option α) (some : is_some x = true)
  : α.
Proof.
  case_eq x.
  + intros y _.
    exact y.
  + intros equ.
    rewrite equ in some.
    now apply is_some_None in some.
Defined.

There is a third approach we can consider: mixing Gallina terms and tactics. This is possible thanks to the ltac:() feature.

Definition from_option'' {α}
    (x : option α) (some : is_some x = true)
  : α :=
  match x as y return x = y -> α with
  | Some x => fun _ => x
  | None => fun equ => ltac:(rewrite equ in some;
                             now apply is_some_None in some)
  end eq_refl.

When Coq encounters ltac:(), it treats it as a hole. It sets up a corresponding goal, and tries to solve it with the supplied tactic.

Conversely, there exists ways to construct terms in Gallina when writing a proof script. Certain tactics take such terms as arguments. Besides, Ltac provides constr:() and uconstr:() which work similarly to ltac:(). The difference between constr:() and uconstr:() is that Coq will try to assign a type to the argument of constr:(), but will leave the argument of uconstr:() untyped.

For instance, consider the following tactic definition.

Tactic Notation "wrap_id" uconstr(x) :=
  let f := uconstr:(fun x => x) in
  exact (f x).

Both x the argument of wrap_id and f the anonymous identity function are not typed. It is only when they are composed together as an argument of exact (which expects a typed argument, more precisely of the type of the goal) that Coq actually tries to type check it.

As a consequence, wrap_id generates a specialized identity function for each specific context.

Definition zero : nat := ltac:(wrap_id 0).

The generated anonymous identity function is fun x : nat => x.

Definition empty_list α : list α := ltac:(wrap_id nil).

The generated anonymous identity function is fun x : list α => x. Besides, we do not need to give more type information about nil. If wrap_id were to be expecting a typed term, we would have to replace nil by [(@nil α)].

Beware the Automation Elephant in the Room

Proofs and computational programs are encoded in Coq as terms, but there is a fundamental difference between them, and it is highlighted by one of the axioms provided by the Coq standard library: proof irrelevance.

Proof irrelevance states that two proofs of the same theorem (i.e., two proof terms which share the same type) are essentially equivalent, and can be substituted without threatening the trustworthiness of the system. From a formal methods point of view, it makes sense. Even if we value “beautiful proofs,” we mostly want correct proofs.

The same reasoning does not apply to computational programs. Two functions of type nat -> nat -> nat are unlikely to be equivalent. For instance, add, mul or sub share the same type, but computes totally different results.

Using tactics such as auto to generate terms which do not live inside Prop is risky, to say the least. For instance,

Definition add (x y : nat) : nat := ltac:(auto).

This works, but it is certainly not what you would expect:

add = fun _ y : nat => y
     : nat -> nat -> nat

That being said, if we keep that in mind, and assert the correctness of the generated programs (either by providing a proof, or by extensively testing it), there is no particular reason not to use Ltac to generate terms when it makes sense.

Dependently typed programming can help you here. If we decorate the return type of a function with the expected properties of the result wrt. the function’s arguments, we can ensure the function is correct, and conversely prevent tactics such as auto to generate “incorrect” terms. Interested readers may refer to the dedicated series on this very website.

Proving Algebraic Datatypes are “Algebraic”

July 12, 2020

Proving Algebraic Datatypes are “Algebraic”

coq

Several programming languages allow programmers to define (potentially recursive) custom types, by composing together existing ones. For instance, in OCaml, one can define lists as follows:

type 'a list =
| Cons of 'a * 'a list
| Nil

This translates in Haskell as

data List a =
  Cons a (List a)
| Nil

In Rust as

enum List<A> {
  Cons(A, Box<List<a>>),
  Nil,
}

Or in Coq as

Inductive list a :=
| cons : a -> list a -> list a
| nil

And so forth.

Each language will have its own specific constructions, and the type systems of OCaml, Haskell, Rust and Coq —to only cite them— are far from being equivalent. That being said, they often share a common “base formalism,” usually (and sometimes abusively) referred to as algebraic datatypes. This expression is used because under the hood any datatype can be encoded as a composition of types using two operators: sum ( $+$ ) and product ( $*$ ) for types.

$a + b$ is the disjoint union of types $a$ and $b$ . Any term of $a$ can be injected into $a + b$ , and the same goes for $b$ . Conversely, a term of $a + b$ can be projected into either $a$ or $b$ .
$a * b$ is the Cartesian product of types $a$ and $b$ . Any term of $a * b$ is made of one term of $a$ and one term of $b$ (think tuples).

For an algebraic datatype, one constructor allows for defining “named tuples,” that is ad hoc product types. Besides, constructors are mutually exclusive: you cannot define the same term using two different constructors. Therefore, a datatype with several constructors is reminiscent of a disjoint union. Coming back to the $\mathrm{list}$ type, under the syntactic sugar of algebraic datatypes, we can define it as

\mathrm{list}_\alpha \equiv \mathrm{unit} + \alpha * \mathrm{list}_\alpha

where $\mathrm{unit}$ models the nil case, and $α * \mathrm{list}_\alpha$ models the cons case.

The set of types which can be defined in a language together with $+$ and $*$ form an “algebraic structure” in the mathematical sense, hence the name. It means the definitions of $+$ and $*$ have to satisfy properties such as commutativity or the existence of neutral elements. In this article, we will prove some of them in Coq. More precisely,

$+$ is commutative, that is $\forall (x, y),\ x + y = y + x$
$+$ is associative, that is $\forall (x, y, z),\ (x + y) + z = x + (y + z)$
$+$ has a neutral element, that is $\exists e_s,\ \forall x,\ x + e_s = x$
$*$ is commutative, that is $\forall (x, y),\ x * y = y * x$
$*$ is associative, that is $\forall (x, y, z),\ (x * y) * z = x * (y * z)$
$*$ has a neutral element, that is $\exists e_p,\ \forall x,\ x * e_p = x$
The distributivity of $+$ and $*$ , that is $\forall (x, y, z),\ x * (y + z) = x * y + x * z$
$*$ has an absorbing element, that is $\exists e_a,\ \forall x, \ x * e_a = e_a$

For the record, the sum ( $+$ ) and prod ( $*$ ) types are defined in Coq as follows:

Inductive sum (A B : Type) : Type :=
| inl : A -> sum A B
| inr : B -> sum A B

Inductive prod (A B : Type) : Type :=
| pair : A -> B -> prod A B

An Equivalence for `Type`

Algebraic structures come with equations expected to be true. This means there is an implicit dependency which is —to my opinion— too easily overlooked: the definition of $=$ . In Coq, = is a built-in relation that states that two terms are “equal” if they can be reduced to the same “hierarchy” of constructors. This is too strong in the general case, and in particular for our study of algebraic structures of Type. It is clear that, to Coq’s opinion, $\alpha + \beta$ is not structurally equal to $\beta + \alpha$ , yet we will have to prove they are “equivalent.”

Introducing `type_equiv`

Since = for Type is not suitable for reasoning about algebraic datatypes, we introduce our own equivalence relation, denoted ==. We say two types $\alpha$ and $\beta$ are equivalent up to an isomorphism when for any term of type $\alpha$ , there exists a counterpart term of type $\beta$ and vice versa. In other words, $\alpha$ and $\beta$ are equivalent if we can exhibit two functions $f$ and $g$ such that:

\forall (x : α),\ x = g(f(x))

\forall (y : β),\ y = f(g(y))

This translates into the following inductive type.

Reserved Notation "x == y" (at level 72).

Inductive type_equiv (α β : Type) : Prop :=
| mk_type_equiv (f : α -> β) (g : β -> α)
                (equ1 : forall (x : α), x = g (f x))
                (equ2 : forall (y : β), y = f (g y))
  : α == β
where "x == y" := (type_equiv x y).

As mentioned earlier, we prove two types are equivalent by exhibiting two functions, and proving these functions satisfy two properties. We introduce a Ltac notation to that end.

Tactic Notation "equiv" "with" uconstr(f) "and" uconstr(g)
  := apply (mk_type_equiv f g).

The tactic equiv with f and g will turn a goal of the form α == β into two subgoals to prove f and g form an isomorphism.

`type_equiv` is an Equivalence

We can prove it by demonstrating it is

Reflexive,
Symmetric, and
Transitive.

`type_equiv` is reflexive

This proof is straightforward. A type $\alpha$ is equivalent to itself because:

\forall (x : α),\ x = id(id(x))

Lemma type_equiv_refl (α : Type) : α == α.
Proof.
  now equiv with (@id α) and (@id α).
Qed.

`type_equiv` is symmetric

If $\alpha = \beta$ , then we know there exists two functions $f$ and $g$ which satisfy the expected properties. We can “swap” them to prove that $\beta == \alpha$ .

Lemma type_equiv_sym {α β} (equ : α == β) : β == α.
Proof.
  destruct equ as [f g equ1 equ2].
  now equiv with g and f.
Qed.

`type_equiv` is transitive

If $\alpha = \beta$ , we know there exists two functions $f_\alpha$ and $g_\beta$ which satisfy the expected properties of type_equiv. Similarly, because $\beta = \gamma$ , we know there exists two additional functions $f_\beta$ and $g_\gamma$ . We can compose these functions together to prove $\alpha = \gamma$ .

As a reminder, composing two functions $f$ and $g$ (denoted by f >>> g thereafter) consists in using the result of $f$ as the input of $g$ .

Infix ">>>" := (fun f g x => g (f x)) (at level 70).

Lemma type_equiv_trans {α β γ} (equ1 : α == β) (equ2 : β == γ)
  : α == γ.
Proof.
  destruct equ1 as [fα gβ equαβ equβα],
           equ2 as [fβ gγ equβγ equγβ].
  equiv with (fα >>> fβ) and (gγ >>> gβ).
  + intros x.
    rewrite <- equβγ.
    now rewrite <- equαβ.
  + intros x.
    rewrite <- equβα.
    now rewrite <- equγβ.
Qed.

Conclusion

The Coq standard library introduces the Equivalence type class. We can provide an instance of this type class for type_equiv, using the three lemmas we have proven in this section.

#[refine]
Instance type_equiv_Equivalence : Equivalence type_equiv :=
  {}.

Proof.
  + intros x.
    apply type_equiv_refl.
  + intros x y.
    apply type_equiv_sym.
  + intros x y z.
    apply type_equiv_trans.
Qed.

Examples

`list`’s canonical form

We now come back to our initial example, given in the introduction of this write-up. We can prove our assertion, that is list α == unit + α * list α.

Lemma list_equiv (α : Type)
  : list α == unit + α * list α.

Proof.
  equiv with (fun x => match x with
                       | [] => inl tt
                       | x :: rst => inr (x, rst)
                       end)
         and (fun x => match x with
                       | inl _ => []
                       | inr (x, rst) => x :: rst
                       end).
  + now intros [| x rst].
  + now intros [[] | [x rst]].
Qed.

`list` is a morphism

This means that if α == β, then list α == list β. We prove this by defining an instance of the Proper type class.

Instance list_Proper
  : Proper (type_equiv ==> type_equiv) list.

Proof.
  add_morphism_tactic.
  intros α β [f g equαβ equβα].
  equiv with (map f) and (map g).
  all: setoid_rewrite map_map; intros l.
  + replace (fun x : α => g (f x))
       with (@id α).
    ++ symmetry; apply map_id.
    ++ apply functional_extensionality.
       apply equαβ.
  + replace (fun x : β => f (g x))
       with (@id β).
    ++ symmetry; apply map_id.
    ++ apply functional_extensionality.
       apply equβα.
Qed.

The use of the Proper type class allows for leveraging hypotheses of the form α == β with the rewrite tactic. I personally consider providing instances of Proper whenever it is possible to be a good practice, and would encourage any Coq programmers to do so.

`nat` is a special purpose `list`

Did you notice? Now, using type_equiv, we can prove it!

Lemma nat_and_list : nat == list unit.
Proof.
  equiv with (fix to_list n :=
                match n with
                | S m => tt :: to_list m
                | _ => []
                end)
         and (fix of_list l :=
                match l with
                | _ :: rst => S (of_list rst)
                | _ => 0
                end).
  + induction x; auto.
  + induction y; auto.
    rewrite <- IHy.
    now destruct a.
Qed.

Non-empty lists

We can introduce a variant of list which contains at least one element by modifying the nil constructor so that it takes one argument instead of none.

Inductive non_empty_list (α : Type) :=
| ne_cons : α -> non_empty_list α -> non_empty_list α
| ne_singleton : α -> non_empty_list α.

We can demonstrate the relation between list and non_empty_list, which reveals an alternative implementation of non_empty_list. More precisely, we can prove that forall (α : Type), non_empty_list α == α * list α. It is a bit more cumbersome, but not that much. We first define the conversion functions, then prove they satisfy the properties expected by type_equiv.

Fixpoint non_empty_list_of_list {α} (x : α) (l : list α)
  : non_empty_list α :=
  match l with
  | y :: rst => ne_cons x (non_empty_list_of_list y rst)
  | [] => ne_singleton x
  end.

#[local]
Fixpoint list_of_non_empty_list {α} (l : non_empty_list α)
  : list α :=
  match l with
  | ne_cons x rst => x :: list_of_non_empty_list rst
  | ne_singleton x => [x]
  end.

Definition list_of_non_empty_list {α} (l : non_empty_list α)
  : α * list α :=
  match l with
  | ne_singleton x => (x, [])
  | ne_cons x rst => (x, list_of_non_empty_list rst)
  end.

Lemma ne_list_list_equiv (α : Type)
  : non_empty_list α == α * list α.

Proof.
  equiv with list_of_non_empty_list
         and (prod_curry non_empty_list_of_list).
  + intros [x rst|x]; auto.
    cbn.
    revert x.
    induction rst; intros x; auto.
    cbn; now rewrite IHrst.
  + intros [x rst].
    cbn.
    destruct rst; auto.
    change (non_empty_list_of_list x (α0 :: rst))
      with (ne_cons x (non_empty_list_of_list α0 rst)).
    replace (α0 :: rst)
      with (list_of_non_empty_list
              (non_empty_list_of_list α0 rst)); auto.
    revert α0.
    induction rst; intros y; [ reflexivity | cbn ].
    now rewrite IHrst.
Qed.

The `sum` Operator

`sum` Is a Morphism

To prove this, we compose together the functions whose existence is implied by $\alpha = \alpha'$ and $\beta = \beta'$ . To that end, we introduce the auxiliary function lr_map.

Definition lr_map_sum {α β α' β'} (f : α -> α') (g : β -> β')
    (x : α + β)
  : α' + β' :=
  match x with
  | inl x => inl (f x)
  | inr y => inr (g y)
  end.

Then, we prove sum is a morphism by defining a Proper instance.

Instance sum_Proper
  : Proper (type_equiv ==> type_equiv ==> type_equiv) sum.
Proof.
  add_morphism_tactic.
  intros α α' [fα gα' equαα' equα'α]
         β β' [fβ gβ' equββ' equβ'β].
  equiv with (lr_map_sum fα fβ)
         and (lr_map_sum gα' gβ').
  + intros [x|y]; cbn.
    ++ now rewrite <- equαα'.
    ++ now rewrite <- equββ'.
  + intros [x|y]; cbn.
    ++ now rewrite <- equα'α.
    ++ now rewrite <- equβ'β.
Qed.

`sum` Is Commutative

Definition sum_invert {α β} (x : α + β) : β + α :=
  match x with
  | inl x => inr x
  | inr x => inl x
  end.

Lemma sum_com {α β} : α + β == β + α.
Proof.
  equiv with sum_invert and sum_invert;
    now intros [x|x].
Qed.

`sum` Is Associative

The associativity of sum is straightforward to prove, and should not pose a particular challenge to prospective readersIf we assume that this article is well written, that is. .

Lemma sum_assoc {α β γ} : α + β + γ == α + (β + γ).
Proof.
  equiv with (fun x =>
                match x with
                | inl (inl x) => inl x
                | inl (inr x) => inr (inl x)
                | inr x => inr (inr x)
                end)
         and (fun x =>
                match x with
                | inl x => inl (inl x)
                | inr (inl x) => inl (inr x)
                | inr (inr x) => inr x
                end).
  + now intros [[x|x]|x].
  + now intros [x|[x|x]].
Qed.

`sum` Has A Neutral Element

We need to find a type $e$ such that $\alpha + e = \alpha$ for any type $\alpha$ (similarly to $x + 0 = x$ for any natural number $x$ ).

Any empty type (that is, a type with no term such as False) can act as the natural element of Type. As a reminder, empty types in Coq are defined with the following syntaxNote that Inductive empty. is erroneous. When the := is omitted, Coq defines an inductive type with one constructor, making such a type equivalent to unit, not False. :

Inductive empty := .

From a high-level perspective, empty being the neutral element of sum makes sense. Because we cannot construct a term of type empty, then α + empty contains exactly the same numbers of terms as α. This is the intuition. Now, how can we convince Coq that our intuition is correct? Just like before, by providing two functions of types:

α -> α + empty
α + empty -> α

The first function is inl, that is one of the constructors of sum.

The second function is trickier to write in Coq, because it comes down to writing a function of type is empty -> α.

Definition from_empty {α} : empty -> α :=
  fun x => match x with end.

It is the exact same trick that allows Coq to encode proofs by contradiction.

If we combine from_empty with the generic function

Definition unwrap_left_or {α β}
    (f : β -> α) (x : α + β)
  : α :=
  match x with
  | inl x => x
  | inr x => f x
  end.

Then, we have everything to prove that α == α + empty.

Lemma sum_neutral (α : Type) : α == α + empty.
Proof.
  equiv with inl and (unwrap_left_or from_empty);
    auto.
  now intros [x|x].
Qed.

The `prod` Operator

This is very similar to what we have just proven for sum, so expect less text for this section.

`prod` Is A Morphism

Definition lr_map_prod {α α' β β'}
    (f : α -> α') (g : β -> β')
  : α * β -> α' * β' :=
  fun x => match x with (x, y) => (f x, g y) end.

Instance prod_Proper
  : Proper (type_equiv ==> type_equiv ==> type_equiv) prod.

Proof.
  add_morphism_tactic.
  intros α α' [fα gα' equαα' equα'α]
         β β' [fβ gβ' equββ' equβ'β].
  equiv with (lr_map_prod fα fβ)
         and (lr_map_prod gα' gβ').
  + intros [x y]; cbn.
    rewrite <- equαα'.
    now rewrite <- equββ'.
  + intros [x y]; cbn.
    rewrite <- equα'α.
    now rewrite <- equβ'β.
Qed.

`prod` Is Commutative

Definition prod_invert {α β} (x : α * β) : β * α :=
  (snd x, fst x).

Lemma prod_com {α β} : α * β == β * α.

Proof.
  equiv with prod_invert and prod_invert;
    now intros [x y].
Qed.

`prod` Is Associative

Lemma prod_assoc {α β γ}
  : α * β * γ == α * (β * γ).

Proof.
  equiv with (fun x =>
                match x with
                | ((x, y), z) => (x, (y, z))
                end)
         and (fun x =>
                match x with
                | (x, (y, z)) => ((x, y), z)
                end).
  + now intros [[x y] z].
  + now intros [x [y z]].
Qed.

`prod` Has A Neutral Element

Lemma prod_neutral (α : Type) : α * unit == α.

Proof.
  equiv with fst and ((flip pair) tt).
  + now intros [x []].
  + now intros.
Qed.

`prod` Has An Absorbing Element *)

And this absorbing element is empty, just like the absorbing element of the multiplication of natural numbers is $0$ (that is, the neutral element of the addition).

Lemma prod_absord (α : Type) : α * empty == empty.
Proof.
  equiv with (snd >>> from_empty)
         and (from_empty).
  + intros [_ []].
  + intros [].
Qed.

`prod` And `sum` Distributivity

Finally, we can prove the distributivity property of prod and sum using a similar approach to prove the associativity of prod and sum.

Lemma prod_sum_distr (α β γ : Type)
  : α * (β + γ) == α * β + α * γ.
Proof.
  equiv with (fun x => match x with
                       | (x, inr y) => inr (x, y)
                       | (x, inl y) => inl (x, y)
                       end)
         and (fun x => match x with
                       | inr (x, y) => (x, inr y)
                       | inl (x, y) => (x, inl y)
                       end).
  + now intros [x [y | y]].
  + now intros [[x y] | [x y]].
Qed.

Bonus: Algebraic Datatypes and Metaprogramming

Algebraic datatypes are very suitable for generating functions, as demonstrated by the automatic deriving of typeclass in Haskell or trait in Rust. Because a datatype can be expressed in terms of sum and prod, you just have to know how to deal with these two constructions to start metaprogramming.

We can take the example of the fold functions. A fold function is a function which takes a container as its argument, and iterates over the values of that container in order to compute a result.

We introduce fold_type INPUT CANON_FORM OUTPUT, a tactic to compute the type of the fold function of the type INPUT, whose “canonical form” (in terms of prod and sum) is CANON_FORM and whose result type is OUTPUT. Interested readers have to be familiar with Ltac.

Ltac fold_args b a r :=
  lazymatch a with
  | unit =>
    exact r
  | b =>
    exact (r -> r)
  | (?c + ?d)%type =>
    exact (ltac:(fold_args b c r) * ltac:(fold_args b d r))%type
  | (b * ?c)%type =>
    exact (r -> ltac:(fold_args b c r))
  | (?c * ?d)%type =>
    exact (c -> ltac:(fold_args b d r))
  | ?a =>
    exact (a -> r)
  end.

Ltac currying a :=
  match a with
  | ?a * ?b -> ?c => exact (a -> ltac:(currying (b -> c)))
  | ?a => exact a
  end.

Ltac fold_type b a r :=
  exact (ltac:(currying (ltac:(fold_args b a r) -> b -> r))).

We use it to compute the type of a fold function for list.

Definition fold_list_type (α β : Type) : Type :=
  ltac:(fold_type (list α) (unit + α * list α)%type β).

fold_list_type =
  fun α β : Type => β -> (α -> β -> β) -> list α -> β
     : Type -> Type -> Type

It is exactly what you could have expected (as match the type of fold_right).

Generating the body of the function is possible in theory, but probably not in Ltac without modifying a bit type_equiv. This could be a nice use case for MetaCoq though.

A Study of Clight and its Semantics

March 20, 2020

A Study of Clight and its Semantics

coq

CompCert is a certified C compiler which comes with a proof of semantics preservation. What this means is the following: the semantics of the C code you write is preserved by CompCert compilation passes up to the generated machine code.

I had been interested in CompCert for quite some times, and ultimately challenged myself to study Clight and its semantics. This write-up is the result of this challenge, written as I was progressing.

Installing CompCert

CompCert has been added to opam, and as a consequence can be very easily used as a library for other Coq developments. A typical use case is for a project to produce Clight (the high-level AST of CompCert), and to benefit from CompCert proofs after that.

Installing CompCert is as easy as

opam install coq-compcert

More precisely, this article uses coq-compcert.3.8.

Once opam terminates, the compcert namespace becomes available. In addition, several binaries are now available if you have correctly set your $PATH environment variable. For instance, clightgen takes a C file as an argument, and generates a Coq file which contains the Clight generated by CompCert.

Problem Statement

Our goal for this first write-up is to prove that the C function

int add (int x, int y) {
    return x + y;
}

returns the expected result, that is x + y. The clightgen tool generates (among other things) the following ASTIt has been modified in order to improve its readability. .

From compcert Require Import Clight Ctypes Clightdefs AST
                             Coqlib Cop.

Definition _x : ident := 1%positive.
Definition _y : ident := 2%positive.

Definition f_add : function :=
  {| fn_return := tint
   ; fn_callconv := cc_default
   ; fn_params := [(_x, tint); (_y, tint)]
   ; fn_vars := []
   ; fn_temps := []
   ; fn_body := Sreturn
                  (Some (Ebinop Oadd
                                (Etempvar _x tint)
                                (Etempvar _y tint)
                                tint))
  |}.

The fields of the function type are pretty self-explanatory (as it is often the case in CompCert’s ASTs as far as I can tell for now).

Identifiers in Clight are (positive) indices. The fn_body field is of type statement, with the particular constructor Sreturn whose argument is of type option expr, and statement and expr look like the two main types to study. The predicates step1 and step2 allow for reasoning about the execution of a function, step-by-step (hence the name). It appears that clightgen generates Clight terms using the function call convention encoded by step2. To reason about a complete execution, it appears that we can use star (from the Smallstep module) which is basically a trace of step. These semantics are defined as predicates (that is, they live in Prop). They allow for reasoning about state transformation, where a state is either

A function call, with a given list of arguments and a continuation
A function return, with a result and a continuation
A statement execution within a function

We import several CompCert modules to manipulate values (in our case, bounded integers).

From compcert Require Import Values Integers.
Import Int.

Putting everything together, the lemma we want to prove about f_add is the following.

Lemma f_add_spec (env : genv)
    (t : Events.trace)
    (m m' : Memory.Mem.mem)
    (v : val) (x y z : int)
    (trace : Smallstep.star step2 env
               (Callstate (Ctypes.Internal f_add)
                          [Vint x; Vint y]
                          Kstop
                          m)
               t
               (Returnstate (Vint z) Kstop m'))
  : z = add x y.

Proof Walkthrough

We introduce a custom inversion tactic which does some clean-up in addition to just perform the inversion.

Ltac smart_inv H :=
  inversion H; subst; cbn in *; clear H.

We can now try to prove our lemma.

Proof.

We first destruct trace, and we rename the generated hypothesis in order to improve the readability of these notes.

  smart_inv trace.
  rename H into Hstep.
  rename H0 into Hstar.

This generates two hypotheses.

Hstep : step1
          env
          (Callstate (Ctypes.Internal f_add)
                     [Vint x; Vint y]
                     Kstop
                     m)
          t1
          s2
Hstar : Smallstep.star
          step2
          env
          s2
          t2
          (Returnstate (Vint z) Kstop m')

In other words, to “go” from a Callstate of f_add to a Returnstate, there is a first step from a Callstate to a state s2, then a succession of steps to go from s2 to a Returnstate.

We consider the single step, in order to determine the actual value of s2 (among other things). To do that, we use smart_inv on Hstep, and again perform some renaming.

  smart_inv Hstep.
  rename le into tmp_env.
  rename e into c_env.
  rename H5 into f_entry.

This produces two effects. First, a new hypothesis is added to the context.

f_entry : function_entry1
            env
            f_add
            [Vint x; Vint y]
            m
            c_env
            tmp_env
            m1

Then, the Hstar hypothesis has been updated, because we now have a more precise value of s2. More precisely, s2 has become

State
  f_add
  (Sreturn
    (Some (Ebinop Oadd
                  (Etempvar _x tint)
                  (Etempvar _y tint)
                  tint)))
  Kstop
  c_env
  tmp_env
  m1

Using the same approach as before, we can uncover the next step.

  smart_inv Hstar.
  rename H into Hstep.
  rename H0 into Hstar.

The resulting hypotheses are

Hstep : step2 env
          (State
             f_add
             (Sreturn
               (Some
                 (Ebinop Oadd
                 (Etempvar _x tint)
                 (Etempvar _y tint)
                 tint)))
             Kstop c_env tmp_env m1) t1 s2
Hstar : Smallstep.star
          step2
          env
          s2
          t0
          (Returnstate (Vint z) Kstop m')

An inversion of Hstep can be used to learn more about its resulting state… So let’s do just that.

  smart_inv Hstep.
  rename H7 into ev.
  rename v0 into res.
  rename H8 into res_equ.
  rename H9 into mem_equ.

The generated hypotheses have become

res : val
ev : eval_expr env c_env tmp_env m1
       (Ebinop Oadd
               (Etempvar _x tint)
               (Etempvar _y tint)
               tint)
       res
res_equ : sem_cast res tint tint m1 = Some v'
mem_equ : Memory.Mem.free_list m1
                               (blocks_of_env env c_env)
            = Some m'0

Our understanding of these hypotheses is the following

The expression _x + _y is evaluated using the c_env environment (and we know thanks to binding the value of _x and _y), and its result is stored in res
res is cast into a tint value, and acts as the result of f_add

The Hstar hypothesis is now interesting

Hstar : Smallstep.star
          step2 env
          (Returnstate v' Kstop m'0) t0
          (Returnstate (Vint z) Kstop m')

It is clear that we are at the end of the execution of f_add (even if Coq generates two subgoals, the second one is not relevant and easy to discard).

  smart_inv Hstar; [| smart_inv H ].

We are making good progress here, and we can focus our attention on the ev hypothesis, which concerns the evaluation of the _x + _y expression. We can simplify it a bit further.

  smart_inv ev; [| smart_inv H].
  rename H4 into fetch_x.
  rename H5 into fetch_y.
  rename H6 into add_op.

In a short-term, the hypotheses fetch_x and fetch_y are the most important.

fetch_x : eval_expr env c_env tmp_env m1 (Etempvar _x tint) v1
fetch_y : eval_expr env c_env tmp_env m1 (Etempvar _y tint) v2

The current challenge we face is to prove that we know their value. At this point, we can have a look at f_entry. This is starting to look familiar: smart_inv, then renaming, etc.

  smart_inv f_entry.
  clear H.
  clear H0.
  clear H1.
  smart_inv H3; subst.
  rename H2 into allocs.

We are almost done. Let’s simplify as much as possible fetch_x and fetch_y. Each time, the smart_inv tactic generates two subgoals, but only the first one is relevant. The second one is not, and can be discarded.

  smart_inv fetch_x; [| inversion H].
  smart_inv H2.
  smart_inv fetch_y; [| inversion H].
  smart_inv H2.

We now know the values of the operands of add. The two relevant hypotheses that we need to consider next are add_op and res_equ. They are easy to read.

add_op : sem_binarith
           (fun (_ : signedness) (n1 n2 : Integers.int)
              => Some (Vint (add n1 n2)))
           (fun (_ : signedness) (n1 n2 : int64)
              => Some (Vlong (Int64.add n1 n2)))
           (fun n1 n2 : Floats.float
              => Some (Vfloat (Floats.Float.add n1 n2)))
           (fun n1 n2 : Floats.float32
              => Some (Vsingle (Floats.Float32.add n1 n2)))
           v1 tint v2 tint m1 = Some res

add_op is the addition of Vint x and Vint y, and its result is res.
```
res_equ : sem_cast res tint tint m1 = Some (Vint z)
```
res_equ is the equation which says that the result of f_add is res, after it has been cast as a tint value.

We can simplify add_op and res_equ, and this allows us to conclude.

  smart_inv add_op.
  smart_inv res_equ.
  reflexivity.
Qed.

Conclusion

The definitions of Clight are straightforward, and the CompCert documentation is very pleasant to read. Understanding Clight and its semantics can be very interesting if you are working on a language that you want to translate into machine code. However, proving some functional properties of a given C snippet using only CompCert can quickly become cumbersome. From this perspective, the VST project is very interesting, as its main purpose is to provide tools to reason about Clight programs more easily.

Release of colorless-themes-0.2

February 15, 2020

Release of `colorless-themes-0.2`

releases

I have tagged and released a new version of colorless-themes . The motivation behind modifying the version number is an important breaking change regarding how the colorless-themes-make macro shall be used.

Before 0.2, the macro was calling deftheme and provide-theme itself. In practices, it meant the actual themes were not using these two functions themselves. It was working great in isolation, but it turns out many tools such as use-package or Peach Melpa (an auto-generated Emacs themes gallery) are relying on the presence of these functions to decide whether a file provides a theme or not. As of now, nordless-theme and lavenderless-theme have been updated accordingly, and they appear on Peach Melpa . Their loading can also be deferred using the :defer keyword of the use-package macro.

if you happen to be a user of colorless-themes, feel free to shoot an email! I would love to hear about your experience using a mostly colorless theme.

A Literate Toolchain To Build This Website

February 4, 2020

A Literate Toolchain To Build This Website

meta literate-programming emacs

Important

What you are reading is actually the rendered version of a Markdown document that was manually “translated” from the Org original document, named Bootstrap.org. Bootstrap.org was probably complete gibberish for anyone who isn’t me. This version was actually heavily reworked to try to fix that.

In any case, the tool-suite described here has not been used to generate this website for a while now.

A literate program is a particular type of software program where code is not directly written in source files, but rather in a text document as code snippets. In essence, literate programming allows for writing in the same place both the software program and its technical documentation.

cleopatra is a “literate toolchain” I have implemented to build this website, and you are currently reading itThis sentence was true when this article was published, but things have changed since then. . That is, cleopatra is both the build system and an article of this website! To achieve this, cleopatra has been written as a collection of org files which can be either “tangled” using Babel or “exported” as a HTML document. Tangling here refers to extract marked code blocks into files.

The page you are currently reading is cleopatra entry point. Its primary purpose is to define two Makefiles —makefile and bootstrap.mk— and the necessary emacs-lisp script to tangle this document.

On the one hand, makefile is the main entrypoint of cleopatra. It serves two purposes:

It initiates a few global variables, and
It provides a rule to tangle this document, that is to update itself and bootstrap.mk.

On the other hand, bootstrap.mk is used to declare the various “generation processes” used to generate this website.

makefile and the emacs-lisp scripts are versioned, because they are necessary to bootstrap cleopatra; but since they are also defined in this document, it means cleopatra can update itself, in some sense. This is to be kept in mind when modifying this document to hastily.

Global Constants and Variables

First, makefile defines several global “constants”As far as I know, make does not support true constant values, It is assumed generation processes will not modify them. . In a nutshell,

$ROOT tells Emacs where the root of your website sources is, so that tangled output filenames can be given relative to it rather than the org files. So for instance, the block_src tangle parameter for Makefile looks like :tangle Makefile, instead of :tangle ../../Makefile.
$CLEODIR tells cleopatra where its sources live. If you place it inside the site/ directory (as it is intended), and you enable the use of org files to author your contents, then cleopatra documents will be part of your website. If you don’t want that, just move the directory outside the site/ directory, and update the $CLEODIR variable accordingly.

For this website, these constants are defined as followsI will use a comment in the first line to recall to which file a given block code is expected to be tangled. .

# makefile:
ROOT := $(shell pwd)
CLEODIR := site/cleopatra

We then introduce two variables to list the output of the generation processes, with two purposes in mind: keeping the .gitignore up-to-date automatically, and providing rules to remove them.

$ARTIFACTS lists the short-term artifacts which can be removed frequently without too much hassle. They will be removed by make clean.
$CONFIGURE lists the long-term artifacts whose generation can be time consuming. They will only be removed by~make cleanall.

# makefile:
ARTIFACTS := build.log
CONFIGURE :=

clean :
	@rm -rf ${ARTIFACTS}

cleanall : clean
	@rm -rf ${CONFIGURE}

Generation processes can declare new build outputs using the += assignement operators. Using another operator will likely provoke an undesirable result.

Tangling Org Documents

cleopatra is a literate program implemented with Org mode, an Emacs major editing mode. We provide the necessary bits to easily tangle Org documents.

The configuration of Babel is done using an emacs lisp script called tangle-org.el whose status is similar to Makefile. It is part of the bootstrap process, and therefore lives “outside” of cleopatra (it is not deleted with make clean for instance). However, it is overwritten when this file is tangled. If you try to modify it and find that cleopatra does not work properly, you should restore it.

;;; tangle-org.el:
(require 'org)
(cd (getenv "ROOT"))
(setq org-confirm-babel-evaluate nil)
(setq org-src-preserve-indentation t)
(add-to-list 'org-babel-default-header-args
             '(:mkdirp . "yes"))
(org-babel-do-load-languages
 'org-babel-load-languages
 '((shell . t)))
(org-babel-tangle)

We define variables that ensure that the $ROOT environment variable is set and tangle-org.el is loaded when using Emacs.

# makefile:
EMACSBIN := emacs
EMACS := ROOT="${ROOT}" ${EMACSBIN}
TANGLE := --batch \
          --load="${ROOT}/scripts/tangle-org.el" \
          2>> build.log

Finally, we introduce a canned recipe to seamlessly tangle a given fileIt was the first time I had used canned recipes, and I don’t think I had the opportunity to re-use it ever again. .

# makefile:
define emacs-tangle =
echo "  tangle  $<"
${EMACS} $< ${TANGLE}
endef

Updating `.gitignore` Automatically

Assuming each generation process correctly defines its $ARTIFACTS and $CONFIGURE variables, we have all the information we need to update .gitignore automatically.

This is done by adding markers in .gitignore to define a region under the control of cleopatra, and writing a script to update said region after each build.

# update-gitignore.sh:
BEGIN_MARKER="# begin generated files"
END_MARKER="# begin generated files"

# remove the previous list of generated files to ignore
sed -i -e "/${BEGIN_MARKER}/,/${END_MARKER}/d" .gitignore
# remove trailing empty lines
sed -i -e :a -e '/^\n*$/{$d;N;};/\n$/ba' .gitignore

# output the list of files to ignore
echo "" >> .gitignore
echo ${BEGIN_MARKER} >> .gitignore
for f in $@; do
    echo "${f}" >> .gitignore
done
echo ${END_MARKER} >> .gitignore

The ignore rule of makefile is defined as follows.

# makefile:
ignore :
	@echo "  update  gitignore"
	@scripts/update-gitignore.sh \
	   ${ARTIFACTS} \
	   ${CONFIGURE}

Bootstrapping

The core purpose of makefile remains to bootstrap the chain of generation processes. This chain is divided into three stages: prebuild, build, and postbuild.

This translates as follows in makefile.

# makefile:
default : postbuild ignore

init :
	@rm -f build.log

prebuild : init

build : prebuild

postbuild : build

.PHONY : init prebuild build postbuild ignore

A generation process in cleopatra is a Makefile which provides rules for these three stages, along with the utilities used by these rules. More precisely, a generation process proc is defined in proc.mk. The rules of proc.mk for each stage are expected to be prefixed by proc-, e.g., proc-prebuild for the prebuild stage.

Eventually, the following dependencies are expected between within the chain of generation processes for every generation process.

prebuild : proc-prebuild
build : proc-build
postbuild : proc-postbuild

proc-build : proc-prebuild
proc-postbuild : proc build

cleopatra is a literate toolchain whose main purpose is to allow me to turn the scripts I wrote to generate my website into blogposts of said website. As such, it allows me to implement the generation processes using Org mode, which means that before being able to start generating HTML files, cleopatra has to tangle the generation processes.

To achieve this, cleopatra relies on a particular behavior of make regarding the include directive. If there exists a rule to generate a Makefile used as an operand of include, make will use this rule to update (if necessary) said Makefile before actually including it.

Therefore, the rules of the following form achieve our ambition of extensibility.

include ${PROC}.mk

prebuild : ${PROC}-prebuild
build : ${PROC}-build
postbuild : ${PROC}-postbuild

${PROC}-prebuild : ${PROC}.mk ${AUX}
${PROC}-build : ${PROC}-prebuild
${PROC}-postbuild : ${PROC}-build

${PROC}.mk ${AUX} &:\
   ${CLEODIR}/${IN}
	@$(emacs-tangle)

CONFIGURE += ${PROC}.mk ${AUX}

.PHONY : ${PROC}-prebuild \
         ${PROC}-build \
         ${PROC}-postbuild

where

$IN is the Org document which contains the generation process code
$PROC is the name of the generation process
$AUX lists the utilities of the generation process tangled from $IN with $PROC.mk

We use &: is used in place of : to separate the target from its dependencies in the “tangle rule.”Yet another obscure Makefile trick I have never encountered again. This tells make that the recipe of this rule generates all these files.

Rather than writing these rules manually for each generation process we want to define, we rely on to noweb of Babel . We call extends the primitive to generate new generation processes.

We derive the rule to tangle bootstrap.mk using extends, using the following Org mode syntax.

#+BEGIN_SRC makefile :noweb yes
# makefile:
<<extends(IN="Bootstrap.org", PROC="bootstrap", AUX="scripts/update-gitignore.sh")>>
#+END_SRC

which gives the following result.

include bootstrap.mk

prebuild : bootstrap-prebuild
build : bootstrap-build
postbuild : bootstrap-postbuild

bootstrap-prebuild : bootstrap.mk scripts/update-gitignore.sh
bootstrap-build : bootstrap-prebuild
bootstrap-postbuild : bootstrap-build

bootstrap.mk scripts/update-gitignore.sh &:\
   ${CLEODIR}/Bootstrap.org
	@$(emacs-tangle)

CONFIGURE += bootstrap.mk scripts/update-gitignore.sh

.PHONY : bootstrap-prebuild \
         bootstrap-build \
         bootstrap-postbuild

These are the last lines of makefile. The rest of the generation processes will be declared in bootstrap.mk.

Generation Processes

In this section, we construct bootstrap.mk by enumerating the generation processes that are currently used to generate the website you are reading.

We recall that each generation process shall

Define proc-prebuild, proc-build, and proc-postbuild
Declare dependencies between stages of generation processes
Declare build outputs (see ARTIFACTS and CONFIGURE)

Theming and Templating

The theme generation process controls the general appearance of the website. More precisely, it introduces the main template used by soupault (main/templates.html), and the main SASS sheet used by this template.

If a generation process produces a set of styles within a specific SASS files, the current approach is

To make this file a dependency of theme-build
To modify style/main.sass in theme to import this file

Eventually, the second step will be automated, but, in the meantime, this customization is mandatory.

Configuring Soupault

The soupault generation configures and run soupault, in order to generate a static website.

If a generation process proc produces files that will eventually be integrated to your website, its proc-build recipe needs to be executed before the soupault-build recipe. This can be enforced by making the dependency explicit to make, i.e.,

soupault-build : proc-build

Eventually, generation processes shall be allowed to produce specific soupault widgets to be integrated into soupault.conf.

Authoring Contents

The fact that cleopatra is a literate program which gradually generates itself was not intended: it is a consequence of my desire to be able to easily use whatever format I so desire for writing my contents, and Org documents in particular.

In the present website, contents can be written in the following format:

HTML Files: This requires no particular set-up, since HTML is the lingua franca of soupault.
Regular Coq files: Coq is a system which allows writing machine-checked proofs, and it comes with a source “prettifier” called coqdoc. Learn more about the generation process for Coq files .
Org documents: Emacs comes with a powerful editing mode called Org mode , and Org documents are really pleasant to work with. Learn more about the generation process for Org documents

I am no longer a PhD. student

January 15, 2019

I am no longer a PhD. student

research

It has been a long journey —4 years, 10 days— but I have completed my PhD on October 25, 2018. The exact title of my PhD thesis is “Specifying and Verifying Hardware-based Security Enforcement Mechanisms ”.

Abstract

In this thesis, we consider a class of security enforcement mechanisms we called Hardware-based Security Enforcement (HSE). In such mechanisms, some trusted software components rely on the underlying hardware architecture to constrain the execution of untrusted software components with respect to targeted security policies. For instance, an operating system which configures page tables to isolate userland applications implements a HSE mechanism.

For a HSE mechanism to correctly enforce a targeted security policy, it requires both hardware and trusted software components to play their parts. During the past decades, several vulnerability disclosures have defeated HSE mechanisms. We focus on the vulnerabilities that are the result of errors at the specification level, rather than implementation errors. In some critical vulnerabilities, the attacker makes a legitimate use of one hardware component to circumvent the HSE mechanism provided by another one. For instance, cache poisoning attacks leverage inconsistencies between cache and DRAM’s access control mechanisms. We call this class of attacks, where an attacker leverages inconsistencies in hardware specifications, compositional attacks.

Our goal is to explore approaches to specify and verify HSE mechanisms using formal methods that would benefit both hardware designers and software developers. Firstly, a formal specification of HSE mechanisms can be leveraged as a foundation for a systematic approach to verify hardware specifications, in the hope of uncovering potential compositional attacks ahead of time. Secondly, it provides unambiguous specifications to software developers, in the form of a list of requirements.

Our contribution is two-fold:

We propose a theory of HSE mechanisms against hardware architecture models. This theory can be used to specify and verify such mechanisms. To evaluate our approach, we propose a minimal model for a single core x86-based computing platform. We use it to specify and verify the HSE mechanism provided by Intel to isolate the code executed while the CPU is in System Management Mode (SMM), a highly privileged execution mode of x86 microprocessors. We have written machine-checked proofs in the Coq proof assistant to that end.
We propose a novel approach inspired by algebraic effects to enable modular verification of complex systems made of interconnected components as a first step towards addressing the challenge posed by the scale of the x86 hardware architecture. This approach is not specific to hardware models, and could also be leveraged to reason about the composition of software components as well. In addition, we have implemented our approach in the Coq theorem prover, and the resulting framework takes advantage of Coq proof automation features to provide general-purpose facilities to reason about components interactions.

Publications

If you are interested, you can have a look at the paper I wrote during my PhD:

SpecCert: Specifying and Verifying Hardware-based Security Enforcement Mechanisms , with Pierre Chifflier, Guillame Hiet and Benjamin Morin, at Formal Methods 2016
Modular Verification of Programs with Effects and Effect Handlers in Coq , with Yann Régis-Gianas, Pierre Chifflier and Guillaume Hiet, at Formal Methods 2018

You can also have a look at the Coq frameworks I have published:

SpecCert on Github (CeCILL-B)
FreeSpec on Github (MPL-2.0)

Discovering Common Lisp with trivial-gamekit

June 17, 2018

Discovering Common Lisp with `trivial-gamekit`

lisp

I always wanted to learn some Lisp dialect. In the meantime, lykan —my Slayers Online clone— begins to take shape. So, of course, my brain got an idea: why not writing a client for lykan in some Lisp dialect?Spoiler alert: this wasn’t the most efficient approach for the lykan project. But it was fun. I asked on Mastodon if there were good game engines for Lisp, and someone told me about trivial-gamekit .

I have no idea if I will manage to implement a decent client using trivial-gamekit, but why not trying? This article is the first of a series about my experiments, discoveries and difficulties. The complete project detailed in this article is available as a gist .

Common Lisp, Quicklisp and `trivial-gamekit`

The trivial-gamekit website lists several requirements. Two are related to Lisp:

Quicklisp
SBCL or CCL

Quicklisp is an experimental package manager for Lisp projects, while SBCL and CCL are two Lisp implementations. I had already installed Clisp , and it took me quite some time to understand my mistake. Fortunately, SBCL is also packaged in ArchLinux.

With a compatible Lisp implementation, installing Quicklisp as a user is straightforward. Following the website instructions is enough. At the end of the process, you will have a new directory ${HOME}/quicklispThe purpose of this directory is similar to the Go workspace . .

Quicklisp is not a native feature of SBCL, and requires a small bit of configurations to be made available automatically. You have to create a file ${HOME}/.sbclrc, with the following content:

(load "~/quicklisp/setup")

There is one final step to be able to use trivial-gamekit.

sbcl --eval '(ql-dist:install-dist "http://bodge.borodust.org/dist/org.borodust.bodge.txt")' \
     --quit

As of June 2018, Quicklisp does not support HTTPS .

Introducing Lysk

Packaging

The first thing I search for when I learn a new language is how projects are organized. From this perspective, trivial-gamekit pointed me directly to Quicklisp

Creating a new Quicklisp project is straightforward. From my understanding, new Quicklisp projects have to be located inside ${HOME}/quicklisp/local-projects. I am not particularly happy with this, but it is not really important.

The current code name of my Lisp game client is lysk.

mkdir ~/quicklisp/local-projects/lysk

Quicklisp packages (systems?) are defined through asd files. I have firstly created lysk.asd as follows:

(asdf:defsystem lysk
  :description "Lykan Game Client"
  :author "lthms"
  :license  "GPLv3"
  :version "0.0.1"
  :serial t
  :depends-on (trivial-gamekit)
  :components ((:file "package")
               (:file "lysk")))

:serial t means that the files detailed in the components field depends on the previous ones. That is, lysk.lisp depends on package.lisp in this case. It is possible to manage files dependencies manually, with the following syntax:

(:file "seconds" :depends-on "first")

I have declared only one dependency: trivial-gamekit. That way, Quicklisp will load it for us.

The first “true” Lisp file we define in our skeleton is package.lisp. Here is its content:

(defpackage :lysk
  (:use :cl)
  (:export run app))

Basically, this means we use two symbols, run and app.

A Game Client

The lysk.lisp file contains the program in itself. My first goal was to obtain the following program: at startup, it shall create a new window in fullscreen, and exit when users release the left button of their mouse. It is worth mentioning that I had to report an issue to the trivial-gamekit upstream in order to make my program work as expected.

While it may sound scary —it suggests trivial-gamekit is a relatively young project— the author has implemented a fix in less than an hour! He also took the time to answer many questions I had when I joined the #lispgames Freenode channel.

Before going any further, let’s have a look at the complete file.

(cl:in-package :lysk)

(gamekit:defgame app () ()
                 (:fullscreen-p 't))

(defmethod gamekit:post-initialize ((app app))
  (gamekit:bind-button :mouse-left :released
                       (lambda () (gamekit:stop))))

(defun run ()
  (gamekit:start 'app))

The first line is some kind of header, to tell Lisp the owner of the file.

The gamekit:defgame function allows for creating a new game application (called app in our case). I ask for a fullscreen window with :fullscreen-p. Then, we use the gamekit:post-initialize hook to bind a handler to the release of the left button of our mouse. This handler is a simple call to gamekit:stop. Finally, we define a new function run which only starts our application.

Pretty straightforward!

Running our Program

To “play” our game, we can start the SBCL REPL.

sbcl --eval '(ql:quickload :lysk)' --eval '(lysk:run)'

A Standalone Executable

It looks like empower a REPL-driven development. That being said, once the development is finished, I don't think I will have a lot of success if I ask my future players to start SBCL to enjoy my game. Fortunately, trivial-gamekit provides a dedicated function to bundle the game as a standalone executable.

Following the advice of the @borodust —the trivial-gamekit author— I created a second package to that end. First, we need to edit the lysk.asd file to detail a second package:

(asdf:defsystem lysk/bundle
  :description "Bundle the Lykan Game Client"
  :author "lthms"
  :license  "GPLv3"
  :version "0.0.1"
  :serial t
  :depends-on (trivial-gamekit/distribution lysk)
  :components ((:file "bundle")))

This second package depends on lysk (our game client) and trivial-gamekit/distribution. The latter provides the deliver function, and we use it in the bundle.lisp file:

(cl:defpackage :lysk.bundle
  (:use :cl)
  (:export deliver))

(cl:in-package :lysk.bundle)

(defun deliver ()
  (gamekit.distribution:deliver :lysk 'lysk:app))

To bundle the game, we can use SBCL from our command line interface.

sbcl --eval "(ql:quickload :lysk/bundle)" \
     --eval "(lysk.bundle:deliver)" \
     --quit

Conclusion

Objectively, there is not much in this article. However, because I am totally new to Lisp, it took me quite some time to get these few lines of code to work together. All being told I think this constitutes a good trivial-gamekit skeleton. Do not hesitate to use it this way.

Thanks again to @borodust , for your time and all your answers!

Appendix: a Makefile

I like Makefile, so here is one to run the game directly, or bundle it.

run:
        @sbcl --eval "(ql:quickload :lysk)" \
              --eval "(lysk:run)"

bundle:
        @echo -en "[ ] Remove old build"
        @rm -rf build/
        @echo -e "\r[*] Remove old build"
        @echo "[ ] Building"
        @sbcl --eval "(ql:quickload :lysk/bundle)" \
              --eval "(lysk.bundle:deliver)" \
              --quit
        @echo "[*] Building"

.PHONY: bundle run

Extensible Type-Safe Error Handling in Haskell

February 4, 2018

Extensible Type-Safe Error Handling in Haskell

haskell

A colleague of mine introduced me to the benefits of error-chain , a crate which aims to implement “consistent error handling” for Rust. I found the overall design pretty convincing, and in his use case, the crate really makes error handling clearer and flexible. I knew Pijul was also using error-chain at that time, but I never had the opportunity to dig more into it.

At the same time, I have read quite a lot about extensible effects in Functional Programming, for an academic article I have submitted to Formal Methods 2018 The odds were in my favor: the aforementioned academic article has been accepted. . In particular, the freer package provides a very nice API to define monadic functions which may use well-identified effects. For instance, we can imagine that Console identifies the functions which may print to and read from the standard output. A function askPassword which displays a prompt and get the user password would have this type signature:

askPassword :: Member Console r => Eff r ()

Compared to IO, Eff allows for meaningful type signatures. It becomes easier to reason about function composition, and you know that a given function which lacks a given effect in its type signature will not be able to use them. As a predictable drawback, Eff can become burdensome to use.

Basically, when my colleague showed me his Rust project and how he was using error-chain, the question popped out. Can we use an approach similar to Eff to implement a Haskell-flavored error-chain?

Spoiler alert: the answer is yes. In this post, I will dive into the resulting API, leaving for another time the details of the underlying implementationFor once, I wanted to write about the result of a project, instead of how it is implemented. . Believe me, there is plenty to say. If you want to have a look already, the current implementation can be found on GitHub .

In this article, I will use several “advanced” GHC pragmas. I will not explain each of them, but I will try to give some pointers for the reader who wants to learn more.

State of the Art

This is not an academic publication, and my goal was primarily to explore the arcane of the Haskell type system, so I might have skipped the proper study of the state of the art. That being said, I have written programs in Rust and Haskell before.

Starting Point

In Rust, Result<T, E> is the counterpart of Either E T in HaskellI wonder if they deliberately choose to swap the two type arguments. . You can use it to model to wrap either the result of a function (T) or an error encountered during this computation (~E~). Both Either and Result are used in order to achieve the same end, that is writing functions which might fail.

On the one hand, Either E is a monad. It works exactly as Maybe (returning an error acts as a shortcut for the rest of the function), but gives you the ability to specify why the function has failed. To deal with effects, the mtl package provides EitherT, a transformer version of Either to be used in a monad stack.

On the other hand, the Rust language provides the ? syntactic sugar, to achieve the same thing. That is, both languages provide you the means to write potentially failing functions without the need to care locally about failure. If your function f uses a function g which might fail, and want to fail yourself if f fails, it becomes trivial.

Out of the box, neither EitherT nor Result is extensible. The functions must use the exact same E, or errors must be converted manually.

Handling Errors in Rust

Rust and the error-chain crate provide several means to overcome this limitation. In particular, it has the Into and From traits to ease the conversion from one error to another. Among other things, the error-chain crate provides a macro to easily define a wrapper around many errors types, basically your own and the one defined by the crates you are using.

I see several drawbacks to this approach. First, it is extensible if you take the time to modify the wrapper type each time you want to consider a new error type. Second, either you can either use one error type or every error type.

However, the error-chain package provides a way to solve a very annoying limitation of Result and Either. When you “catch” an error, after a given function returns its result, it can be hard to determine from where the error is coming from. Imagine you are parsing a very complicated source file, and the error you get is SyntaxError with no additional context. How would you feel?

error-chain solves this by providing an API to construct a chain of errors, rather than a single value.

my_function().chain_err(|| "a message with some context")?;

The chain_err function makes it easier to replace a given error in its context, leading to be able to write more meaningful error messages for instance.

The `ResultT` Monad

The ResultT is an attempt to bring together the extensible power of Eff and the chaining of errors of chain_err. I will admit that, for the latter, the current implementation of ResultT is probably less powerful, but to be honest I mostly cared about the “extensible” thing, so it is not very surprising.

This monad is an alternative to neither Monad Stacks a la mtl nor to the Eff monad. In its current state, it aims to be a more powerful and flexible version of EitherT.

Parameters

As often in Haskell, the ResultT monad can be parameterized in several ways.

data ResultT msg (err :: [*]) m a

msg is the type of messages you can stack to provide more context to error handling
err is a row of errorsYou might have noticed err is of kind [*]. To write such a thing, you will need the DataKinds GHC pragmas. , it basically describes the set of errors you will eventually have to handle
m is the underlying monad stack of your application, knowing that ResultT is not intended to be stacked itself
a is the expected type of the computation result

`achieve` and `abort`

The two main monadic operations which comes with ~ResultT~ are ~achieve~ and ~abort~. The former allows for building the context, by stacking so-called messages which describe what you want to do. The latter allows for bailing on a computation and explaining why.

achieve :: (Monad m)
        => msg
        -> ResultT msg err m a
        -> ResultT msg err m a

achieve should be used for do blocks. You can use <?> to attach a contextual message to a given computation.

The type signature of abort is also interesting, because it introduces the Contains typeclass (e.g., it is equivalent to Member for Eff).

abort :: (Contains err e, Monad m)
      => e
      -> ResultT msg err m a

This reads as follows: “you can abort with an error of type e if and only if the row of errors err contains the type e.”

For instance, imagine we have an error type FileError to describe filesystem-related errors. Then, we can imagine the following function:

readContent :: (Contains err FileError, MonadIO m)
            => FilePath
            -> ResultT msg err m String

We could leverage this function in a given project, for instance, to read its configuration files (for the sake of the example, it has several configuration files). This function can use its own type to describe ill-formed description (ConfigurationError).

parseConfiguration :: (Contains err ConfigurationError, MonadIO m)
                   => String
                   -> String
                   -> ResultT msg err m Configuration

To avoid repeating Contains when the row of errors needs to contains several elements, we introduce :<If you are confused by :<, it is probably because you were not aware that the TypeOperators GHC pragma was a thing. (read subset or equal):

getConfig :: ( '[FileError, ConfigurationError] :< err
             , MonadIO m)
             => ResultT String err m Configuration
getConfig = do
  achieve "get configuration from ~/.myapp directory" $ do
    f1 <- readContent "~/.myapp/init.conf"
              <?> "fetch the main configuration"
    f2 <- readContent "~/.myapp/net.conf"
              <?> "fetch the net-related configuration"

    parseConfiguration f1 f2

You might see, now, why I say ~ResultT~ is extensible. You can use two functions with totally unrelated errors, as long as the caller advertises that with ~Contains~ or ~:<~.

Recovering by Handling Errors

Monads are traps, you can only escape them by playing with their rules. ResultT comes with runResultT.

runResultT :: Monad m => ResultT msg '[] m a -> m a

This might be surprising: we can only escape out from the ResultT if we do not use any errors at all. That is, ResultT forces us to handle errors before calling runResultT.

ResultT provides several functions prefixed by recover. Their type signatures can be a little confusing, so we will dive into the simpler one:

recover :: forall e m msg err a.
           (Monad m)
        => ResultT msg (e ': err) m a
        -> (e -> [msg] -> ResultT msg err m a)
        -> ResultT msg err m a

recover allows for removing an error type from the row of errors, To do that, it requires to provide an error handler to determine what to do with the error raised during the computation and the stack of messages at that time. Using recover, a function may use more errors than advertised in its type signature, but we know by construction that in such a case, it handles these errors so that it is transparent for the function user. The type of the handler is e -> [msg] -> ResultT msg err m a, which means the handler can raise errors if required.

recoverWhile msg is basically a synonym for achieve msg $ recover. recoverMany allows for doing the same with a row of errors, by providing as many functions as required. Finally, recoverManyWith simplifies recoverMany: you can provide only one function tied to a given typeclass, on the condition that the handling errors implement this typeclass.

Using recover and its siblings often require to help a bit the Haskell type system, especially if we use lambdas to define the error handlers. Doing that is usually achieved with the Proxy a dataype (where a is a phantom type). I would rather use the TypeApplications pragmaThe TypeApplications pragmas is probably one of my favorites. When I use it, it feels almost like if I were writing a Coq document. .

recoverManyWith @[FileError, NetworkError] @DescriptiveError
    (do x <- readFromFile f
        y <- readFromNetwork socket
        printToStd x y)
    printErrorAndStack

The DecriptiveError typeclass can be seen as a dedicated Show, to give textual representation of errors. It is inspired by the macros of error_chain.

We can start from an empty row of errors, and allows ourselves to use more errors thanks to the recover* functions.

`cat` in Haskell using `ResultT`

ResultT only cares about error handling. The rest of the work is up to the underlying monad m. That being said, nothing forbids us to provide fine-grained API, e.g., for Filesystem-related functions. From an error handling perspective, the functions provided by Prelude (the standard library of Haskell) are pretty poor, and the documentation is not really precise regarding the kind of error we can encounter while using it.

In this section, I will show you how we can leverage ResultT to (i) define an error-centric API for basic file management functions and (ii) use this API to implement a cat-like program which read a file and print its content in the standard output.

(A Lot Of) Error Types

We could have one sum type to describe in the same place all the errors we can find, and later use the pattern matching feature of Haskell to determine which one has been raised. The thing is, this is already the job done by the row of errors of ~ResultT~. Besides, this means that we could raise an error for being not able to write something into a file in a function which /opens/ a file.

Because ~ResultT~ is intended to be extensible, we should rather define several types, so we can have a fine-grained row of errors. Of course, too many types will become burdensome, so this is yet another time where we need to find the right balance.

newtype AlreadyInUse = AlreadyInUse FilePath
newtype DoesNotExist = DoesNotExist FilePath
data AccessDeny = AccessDeny FilePath IO.IOMode
data EoF = EoF
data IllegalOperation = IllegalRead | IllegalWrite

To be honest, this is a bit too much for the real life, but we are in a blog post here, so we should embrace the potential of ResultT.

Filesystem API

By reading the System.IO documentation, we can infer what our functions type signatures should look like. I will not discuss their actual implementation in this article, as this requires me to explain how IO deals with errors itself (and this article is already long enough to my taste). You can have a look at this gist if you are interested.

openFile :: ( '[AlreadyInUse, DoesNotExist, AccessDeny] :< err
            , MonadIO m)
         => FilePath -> IOMode -> ResultT msg err m Handle

getLine :: ('[IllegalOperation, EoF] :< err, MonadIO m)
        => IO.Handle
        -> ResultT msg err m Text

closeFile :: (MonadIO m)
          => IO.Handle
          -> ResultT msg err m ()

Implementing `cat`

We can use the ResultT monad, its monadic operations and our functions to deal with the file system in order to implement a cat-like program. I tried to comment on the implementation to make it easier to follow.

cat :: FilePath -> ResultT String err IO ()
cat path =
  -- We will try to open and read this file to mimic
  -- `cat` behaviour.
  -- We advertise that in case something goes wrong
  -- the process.
  achieve ("cat " ++ path) $ do
    -- We will recover from a potential error,
    -- but we will abstract away the error using
    -- the `DescriptiveError` typeclass. This way,
    -- we do not need to give one handler by error
    -- type.
    recoverManyWith @[Fs.AlreadyInUse, Fs.DoesNotExist, Fs.AccessDeny, Fs.IllegalOperation]
                    @(Fs.DescriptiveError)
      (do f <- Fs.openFile path Fs.ReadMode
          -- `repeatUntil` works like `recover`, except
          -- it repeats the computation until the error
          -- actually happpens.
          -- I could not have used `getLine` without
          -- `repeatUntil` or `recover`, as it is not
          -- in the row of errors allowed by
          -- `recoverManyWith`.
          repeatUntil @(Fs.EoF)
              (Fs.getLine f >>= liftIO . print)
              (\_ _ -> liftIO $ putStrLn "%EOF")
          closeFile f)
      printErrorAndStack
    where
      -- Using the `DescriptiveError` typeclass, we
      -- can print both the stack of Strings which form
      -- the context, and the description of the generic
      -- error.
      printErrorAndStack e ctx = do
        liftIO . putStrLn $ Fs.describe e
        liftIO $ putStrLn "stack:"
        liftIO $ print ctx

The type signature of cat teaches us that this function handles any error it might encounter. This means we can use it anywhere we want: both in another computation inside ResultT which might raise errors completely unrelated to the file system, or we can use it with runResultT, escaping the ResultT monad (only to fall into the IO monad, but this is another story).

Monad Transformers are a Great Abstraction

July 15, 2017

Monad Transformers are a Great Abstraction

haskell

Caution

Time has passed since the publication of this article. Whether or not I agree with its conclusions is an open question. Monad Transformers are a great abstraction, but nowadays I would most certainly choose another approach.

Monads are hard to get right. I think it took me around a year of Haskelling to feel like I understood them. The reason is, to my opinion, there is not such thing as the Monad. It is even the contrary. When someone asks me how I would define Monads in only a few words, I say monads are a convenient formalism to chain specific computations. Once I’ve got that, I started noticing “monadic construction” everywhere, from the Rust ? operator to the Elixir with keyword .

Haskell often uses another concept above Monads: Monad Transformers. This allows you to work not only with one Monad, but rather a stack. Each Monad brings its own properties and you can mix them into your very own one. That you can’t have in Rust or Elixir, but it works great in Haskell. Unfortunately, it is not an easy concept and it can be hard to understand. This article is not an attempt to do so, but rather a postmortem review of one situation where I found them extremely useful. If you think you have understood how they work, but don’t see the point yet, you might find here a beginning of the answer.

Recently, I ran into a very good example of why Monad Transformers worth it. I have been working on a project called ogma for a couple years now. In a nutshell, I want to build “a tool” to visualize in time and space a storytelling. We are not here just yet, but, in the meantime, I have written a software called celtchar to build a novel from a list of files. One of its newest features is the choice of language, and by extension, the typographic rules. This information is read from a configuration file very early in the program flow. Unfortunately, its use comes much later, after several function calls.

In Haskell, you deal with that kind of challenge by relying on the Reader Monad. It carries an environment in a transparent way. The only thing is, I was already using the State Monad to carry the computation result. But that’s not an issue with the Monad Transformers.

-type Builder = StateT Text IO
+type Builder = StateT Text (ReaderT Language IO)

As you may have already understood, I wasn't using the “raw” State Monad, but rather the transformer version StateT. The underlying Monad was IO, because I needed to be able to read some files from the file system. By replacing IO by ReaderT Language IO, I basically fixed my “carry the variable to the correct function call easily” problem.

Retrieving the chosen language is as simple as:

getLanguage :: Builder Language
getLanguage = lift ask

And that was basically it.

Now, my point is not that Monad Transformers are the ultimate beast we will have to tame once and then everything will be shiny and easyIt is amusing to see Past Me being careful here. . There are a lot of other ways to achieve what I did with my Builder stack. For instance, in an OO language, I probably would have to add a new class member to my Builder class and I would have done basically the same thing.

However, there is something I really like about this approach: the Builder type definition gives you a lot of useful information already. Both the State and Reader Monads have a well-established semantics most Haskellers will understand in a glance. A bit of documentation won’t hurt, but I suspect it is not as necessary as one could expect. Moreover, the presence of the IO Monad tells everyone using the Builder Monad might cause I/O.

Rewriting in Coq

May 13, 2017

Rewriting in Coq

coq

I have to confess something. In the codebase of SpecCert lies a shameful secret, which takes the form of a set of unnecessary axioms.

I thought I couldn’t avoid them at first, but it was before I heard about “generalized rewriting,” setoids and morphisms. Now, I know the truth, and I will have to update SpecCert eventually. But, in the meantime, let me try to explain how it is possible to rewrite a term in a proof using an ad hoc equivalence relation and, when necessary, a proper morphism.

Case Study: Gate System

Now, why would anyone want such a thing as “generalized rewriting” when the rewrite tactic works just fine.

The thing is: it does not in some cases. To illustrate my statement, we will consider the following definition of a gate in Coq:

Record Gate :=
  { open:           bool
  ; lock:           bool
  ; lock_is_close:  lock = true -> open = false
  }.

According to this definition, a gate can be either open or closed. It can also be locked, but if it is, it cannot be open at the same time. To express this constraint, we embed the appropriate proposition inside the Record. By doing so, we know for sure that we will never meet an ill-formed Gate instance. The Coq engine will prevent it, because to construct a gate, one will have to prove the lock_is_close predicate holds.

The program attribute makes it easy to work with embedded proofs. For instance, defining the ”open gate” is as easy as:

Require Import Coq.Program.Tactics.

#[program]
Definition open_gate :=
  {| open := true
   ; lock := false
   |}.

Under the hood, program proves what needs to be proven, that is the lock_is_close proposition. Just have a look at its output:

open_gate has type-checked, generating 1 obligation(s)
Solving obligations automatically...
open_gate_obligation_1 is defined
No more obligations remaining
open_gate is defined

In this case, using Program is a bit like cracking a nut with a sledgehammer. We can easily do it ourselves using the refine tactic.

Definition open_gate': Gate.
  refine ({| open := true
           ; lock := false
          |}).
  intro Hfalse.
  discriminate Hfalse.
Defined.

`Gate` Equality

What does it mean for two gates to be equal? Intuitively, we know they have to share the same states (open and lock is our case).

Leibniz Equality Is Too Strong

When you write something like a = b in Coq, the = refers to the eq function and this function relies on what is called the Leibniz Equality: x and y are equal iff every property on A which is true of x is also true of y.

As for myself, when I first started to write some Coq code, the Leibniz Equality was not really something I cared about and I tried to prove something like this:

Lemma open_gates_are_equal (g g': Gate)
    (equ : open g = true) (equ' : open g' = true)
  : g = g'.

Basically, it means that if two doors are open, then they are equal. That made sense to me, because by definition of Gate, a locked door is closed, meaning an open door cannot be locked.

Here is an attempt to prove the open_gates_are_equal lemma.

Proof.
  assert (forall g, open g = true -> lock g = false). {
    intros [o l h] equo.
    cbn in *.
    case_eq l; auto.
    intros equl.
    now rewrite (h equl) in equo.
  }
  assert (lock g = false) by apply (H _ equ).
  assert (lock g' = false) by apply (H _ equ').
  destruct g; destruct g'; cbn in *; subst.

The term to prove is now:

{| open := true; lock := false; lock_is_close := lock_is_close0 |} =
{| open := true; lock := false; lock_is_close := lock_is_close1 |}

The next tactic I wanted to use reflexivity, because I'd basically proven open g = open g' and lock g = lock g', which meets my definition of equality at that time.

Except Coq wouldn’t agree. See how it reacts:

Unable to unify "{| open := true; lock := false; lock_is_close := lock_is_close1 |}"
  with "{| open := true; lock := false; lock_is_close := lock_is_close0 |}".

It cannot unify the two records. More precisely, it cannot unify lock_is_close1 and lock_is_close0. So we abort and try something else.

Abort.

Ah-Hoc Equivalence Relation

This is a familiar pattern. Coq cannot guess what we have in mind. Giving a formal definition of “our equality” is fortunately straightforward.

Definition gate_eq
           (g g': Gate)
  : Prop :=
  open g = open g' /\ lock g = lock g'.

Because “equality” means something very specific in Coq, we won't say “two gates are equal” anymore, but “two gates are equivalent.” That is, gate_eq is an equivalence relation. But “equivalence relation” is also something very specific. For instance, such relation needs to be symmetric (R x y -> R y x), reflexive (R x x) and transitive (R x y -> R y z -> R x z).

Require Import Coq.Classes.Equivalence.

#[program]
Instance Gate_Equivalence
  : Equivalence gate_eq.

Next Obligation.
  split; reflexivity.
Defined.

Next Obligation.
  intros g g' [Hop Hlo].
  symmetry in Hop; symmetry in Hlo.
  split; assumption.
Defined.

Next Obligation.
  intros g g' g'' [Hop Hlo] [Hop' Hlo'].
  split.
  + transitivity (open g'); [exact Hop|exact Hop'].
  + transitivity (lock g'); [exact Hlo|exact Hlo'].
Defined.

Afterwards, the symmetry, reflexivity and transitivity tactics also works with gate_eq, in addition to eq. We can try again to prove the open_gate_are_the_same lemma and it will workI know I should have proven an intermediate lemma to avoid code duplication. Sorry about that, it hurts my eyes too. .

Lemma open_gates_are_the_same:
  forall (g g': Gate),
    open g = true
    -> open g' = true
    -> gate_eq g g'.
Proof.
  induction g; induction g'.
  cbn.
  intros H0 H2.
  assert (lock0 = false).
  + case_eq lock0; [ intro H; apply lock_is_close0 in H;
                     rewrite H0 in H;
                     discriminate H
                   | reflexivity
                   ].
  + assert (lock1 = false).
    * case_eq lock1; [ intro H'; apply lock_is_close1 in H';
                       rewrite H2 in H';
                       discriminate H'
                     | reflexivity
                     ].
    * subst.
      split; reflexivity.
Qed.

Equivalence Relations and Rewriting

So here we are, with our ad hoc definition of gate equivalence. We can use symmetry, reflexivity and transitivity along with gate_eq and it works fine because we have told Coq gate_eq is indeed an equivalence relation for Gate.

Can we do better? Can we actually use rewrite to replace an occurrence of g by an occurrence of g’ as long as we can prove that gate_eq g g’? The answer is “yes”, but it will not come for free.

Before moving forward, just consider the following function:

Require Import Coq.Bool.Bool.

Program Definition try_open
        (g: Gate)
  : Gate :=
  if eqb (lock g) false
  then {| lock := false
        ; open := true
       |}
  else g.

It takes a gate as an argument and returns a new gate. If the former is not locked, the latter is open. Otherwise the argument is returned as is.

Lemma gate_eq_try_open_eq
  : forall (g g': Gate),
    gate_eq g g'
    -> gate_eq (try_open g) (try_open g').
Proof.
  intros g g' Heq.
Abort.

What we could have wanted to do is: rewrite Heq. Indeed, g and g’ “are the same” (gate_eq g g’), so, of course, the results of try_open g and try_open g’ have to be the same. Except...

Error: Tactic failure: setoid rewrite failed: Unable to satisfy the following constraints:
UNDEFINED EVARS:
 ?X49==[g g' Heq |- relation Gate] (internal placeholder) {?r}
 ?X50==[g g' Heq (do_subrelation:=Morphisms.do_subrelation)
         |- Morphisms.Proper (gate_eq ==> ?X49@{__:=g; __:=g'; __:=Heq}) try_open] (internal placeholder) {?p}
 ?X52==[g g' Heq |- relation Gate] (internal placeholder) {?r0}
 ?X53==[g g' Heq (do_subrelation:=Morphisms.do_subrelation)
         |- Morphisms.Proper (?X49@{__:=g; __:=g'; __:=Heq} ==> ?X52@{__:=g; __:=g'; __:=Heq} ==> Basics.flip Basics.impl) eq]
         (internal placeholder) {?p0}
 ?X54==[g g' Heq |- Morphisms.ProperProxy ?X52@{__:=g; __:=g'; __:=Heq} (try_open g')] (internal placeholder) {?p1}

What Coq is trying to tell us here —in a very poor manner, I’d say— is actually pretty simple. It cannot replace g by g’ because it does not know if two equivalent gates actually give the same result when passed as the argument of try_open. This is actually what we want to prove, so we cannot use rewrite just yet, because it needs this result so it can do its magic. Chicken and egg problem.

In other words, we are making the same mistake as before: not telling Coq what it cannot guess by itself.

The rewrite tactic works out of the box with the Coq equality (eq, or most likely =) because of the Leibniz Equality: x and y are equal iff every property on A which is true of x is also true of y

This is a pretty strong property, and one that a lot of equivalence relations do not have. Want an example? Consider the relation R over A such that forall x and y, R x y holds true. Such a relation is reflexive, symmetric and reflexive. Yet, there is very little chance that given a function f : A -> B and R’ an equivalence relation over B, R x y -> R' (f x) (f y). Only if we have this property, we would know that we could rewrite f x by f y, e.g., in R' z (f x). Indeed, by transitivity of R’, we can deduce R' z (f y) from R' z (f x) and R (f x) (f y).

If R x y -> R' (f x) (f y), then f is a morphism because it preserves an equivalence relation. In our previous case, A is Gate, R is gate_eq, f is try_open and therefore B is Gate and R’ is gate_eq. To make Coq aware that try_open is a morphism, we can use the following syntax: *)

#[local]
Open Scope signature_scope.

Require Import Coq.Classes.Morphisms.

#[program]
Instance try_open_Proper
  : Proper (gate_eq ==> gate_eq) try_open.

This notation is actually more generic because you can deal with functions that take more than one argument. Hence, given g : A -> B -> C -> D, R (respectively R’, R’’ and R’’’) an equivalent relation of A (respectively B, C and D), we can prove f is a morphism as follows:

Add Parametric Morphism: (g)
    with signature (R) ==> (R') ==> (R'') ==> (R''')
      as <name>.

Back to our try_open morphism. Coq needs you to prove the following goal:

1 subgoal, subgoal 1 (ID 50)

  ============================
  forall x y : Gate, gate_eq x y -> gate_eq (try_open x) (try_open y)

Here is a way to prove that:

Next Obligation.
  intros g g' Heq.
  assert (gate_eq g g') as [Hop Hlo] by (exact Heq).
  unfold try_open.
  rewrite <- Hlo.
  destruct (bool_dec (lock g) false) as [Hlock|Hnlock]; subst.
  + rewrite Hlock.
    split; cbn; reflexivity.
  + apply not_false_is_true in Hnlock.
    rewrite Hnlock.
    cbn.
    exact Heq.
Defined.

Now, back to our gate_eq_try_open_eq, we now can use rewrite and reflexivity.

Require Import Coq.Setoids.Setoid.

Lemma gate_eq_try_open_eq
  : forall (g g': Gate),
    gate_eq g g'
    -> gate_eq (try_open g) (try_open g').
Proof.
  intros g g' Heq.
  rewrite Heq.
  reflexivity.
Qed.

We did it! We actually rewrite g as g’, even if we weren’t able to prove g = g’.

Implementing Strongly-Specified Functions with the Program Framework

January 1, 2017

Implementing Strongly-Specified Functions with the `Program` Framework

coq

The Theory

If I had to explain Program, I would say Program is the heir of the refine tactic. It gives you a convenient way to embed proofs within functional programs that are supposed to fade away during code extraction. But what do I mean when I say "embed proofs" within functional programs? I found two ways to do it.

Invariants

First, we can define a record with one or more fields of type Prop. By doing so, we can constrain the values of other fields. Put another way, we can specify invariant for our type. For instance, in SpecCert , I have defined the memory controller's SMRAMC register as follows:

Record SmramcRegister := {
  d_open: bool;
  d_lock: bool;
  lock_is_close: d_lock = true -> d_open = false;
}.

So lock_is_closed is an invariant I know each instance of SmramcRegister will have to comply with, because every time I will construct a new instance, I will have to prove lock_is_closed holds true. For instance:

Definition lock (reg: SmramcRegister)
  : SmramcRegister.
  refine ({| d_open := false; d_lock := true |}).

Coq leaves us this goal to prove.

reg : SmramcRegister
============================
true = true -> false = false

This sound reasonable enough.

Proof.
  trivial.
Defined.

We have seen in my previous article about strongly specified functions that mixing proofs and regular terms may lead to cumbersome code.

From that perspective, Program helps. Indeed, the lock function can also be defined as follows:

From Coq Require Import Program.

#[program]
Definition lock' (reg: SmramcRegister)
  : SmramcRegister :=
  {| d_open := false
   ; d_lock := true
   |}.

Pre and Post Conditions

Another way to "embed proofs in a program" is by specifying pre- and post-conditions for its component. In Coq, this is done using sigma types.

On the one hand, a precondition is a proposition a function input has to satisfy in order for the function to be applied. For instance, a precondition for head : forall {a}, list a -> a the function that returns the first element of a list l requires l to contain at least one element. We can write that using a sigma-type. The type of head then becomes forall {a} (l: list a | l <> []) : a.

On the other hand, a post condition is a proposition a function output has to satisfy in order for the function to be correctly implemented. In this way, head should in fact return the first element of l and not something else.

Program makes writing this specification straightforward.

#[program]
Definition head {a} (l : list a | l <> [])
  : { x : a | exists l', x :: l' = l }.

We recall that because { l: list a | l <> [] } is not the same as list a, in theory we cannot just compare l with x :: l' (we need to use proj1_sig). One advantage of Program is to deal with it using an implicit coercion.

Note that for the type inference to work as expected, the unwrapped value (here, x :: l') needs to be the left operand of =.

Now that head have been specified, we have to implement it.

#[program]
Definition head {a} (l: list a | l <> [])
  : { x : a | exists l', cons x l' = l } :=
  match l with
  | x :: l' => x
  | [] => !
  end.

Next Obligation.
  exists l'.
  reflexivity.
Qed.

I want to highlight several things here:

We return x (of type a) rather than a sigma-type, then Program is smart enough to wrap it. To do so, it tries to prove the post condition and because it fails, we have to do it ourselves (this is the Obligation we solve after the function definition.)
The [] case is absurd regarding the precondition, we tell Coq that using the bang (!) symbol.

We can have a look at the extracted code:

(** val head : 'a1 list -> 'a1 **)
let head = function
| Nil -> assert false (* absurd case *)
| Cons (a, _) -> a

The implementation is pretty straightforward, but the pre- and post conditions have faded away. Also, the absurd case is discarded using an assertion. This means one thing: [head] should not be used directly from the Ocaml world. "Interface" functions have to be total. *)

The Practice

From Coq Require Import Lia.

I have challenged myself to build a strongly specified library. My goal was to define a type vector : nat -> Type -> Type such as vector a n is a list of n instance of a.

Inductive vector (a : Type) : nat -> Type :=
| vcons {n} : a -> vector a n -> vector a (S n)
| vnil : vector a O.

Arguments vcons [a n] _ _.
Arguments vnil {a}.

I had three functions in mind: take, drop and extract. I learned a few lessons. My main takeaway remains: do not use sigma types, Program and dependent types together. From my point of view, Coq is not yet ready for this. Maybe it is possible to make those three work together, but I have to admit I did not find out how. As a consequence, my preconditions are defined as extra arguments.

To be able to specify the post conditions of my three functions and some others, I first defined nth to get the nth element of a vector.

My first attempt to write nth was a failure.

#[program]
Fixpoint nth {a n}
    (v : vector a n) (i : nat) {struct v}
  : option a :=
  match v, i with
  | vcons x _, O => Some x
  | vcons x r, S i => nth r i
  | vnil, _ => None
  end.

raised an anomaly.

#[program]
Fixpoint nth {a n}
    (v : vector a n) (i : nat) {struct v}
  : option a :=
  match v with
  | vcons x r =>
    match i with
    | O => Some x
    | S i => nth r i
    end
  | vnil => None
  end.

With nth, it is possible to give a very precise definition of take:

#[program]
Fixpoint take {a n}
    (v : vector a n) (e : nat | e <= n)
  : { u : vector a e | forall i : nat,
        i < e -> nth u i = nth v i } :=
  match e with
  | S e' => match v with
            | vcons x r => vcons x (take r e')
            | vnil => !
            end
  | O => vnil
  end.

Next Obligation.
  now apply le_S_n.
Defined.

Next Obligation.
  induction i.
  + reflexivity.
  + apply e0.
    now apply Lt.lt_S_n.
Defined.

Next Obligation.
  now apply PeanoNat.Nat.nle_succ_0 in H.
Defined.

Next Obligation.
  now apply PeanoNat.Nat.nlt_0_r in H.
Defined.

As a side note, I wanted to define the post condition as follows: { v': vector A e | forall (i : nat | i < e), nth v' i = nth v i }. However, this made the goals and hypotheses become very hard to read and to use. Sigma types in sigma types: not a good idea.

(** val take : 'a1 vector -> nat -> 'a1 vector **)

let rec take v = function
| O -> Vnil
| S e' ->
  (match v with
   | Vcons (_, x, r) -> Vcons (e', x, (take r e'))
   | Vnil -> assert false (* absurd case *))

Then I could tackle drop in a very similar manner:

#[program]
Fixpoint drop {a n}
    (v : vector a n) (b : nat | b <= n)
  : { v': vector a (n - b) | forall i,
        i < n - b -> nth v' i = nth v (b + i) } :=
  match b with
  | 0 => v
  | S n => (match v with
           | vcons _ r => (drop r n)
           | vnil => !
           end)
  end.

Next Obligation.
  now rewrite <- Minus.minus_n_O.
Defined.

Next Obligation.
  induction n;
    rewrite <- eq_rect_eq;
    reflexivity.
Defined.

Next Obligation.
  now apply le_S_n.
Defined.

Next Obligation.
  now apply PeanoNat.Nat.nle_succ_0 in H.
Defined.

The proofs are easy to write, and the extracted code is exactly what one might want it to be:

(** val drop : 'a1 vector -> nat -> 'a1 vector **)
let rec drop v = function
| O -> v
| S n ->
  (match v with
   | Vcons (_, _, r) -> drop r n
   | Vnil -> assert false (* absurd case *))

But Program really shone when it comes to implementing extract. I just had to combine take and drop. *)

#[program]
Definition extract {a n} (v : vector a n)
    (e : nat | e <= n) (b : nat | b <= e)
  : { v': vector a (e - b) | forall i,
        i < (e - b) -> nth v' i = nth v (b + i) } :=
  take (drop v b) (e - b).


Next Obligation.
  transitivity e; auto.
Defined.

Next Obligation.
  now apply PeanoNat.Nat.sub_le_mono_r.
Defined.

Next Obligation.
  destruct drop; cbn in *.
  destruct take; cbn in *.
  rewrite e1; auto.
  rewrite <- e0; auto.
  lia.
Defined.

The proofs are straightforward because the specifications of drop and take are precise enough, and we do not need to have a look at their implementations. The extracted version of extract is as clean as we can anticipate.

(** val extract : 'a1 vector -> nat -> nat -> 'a1 vector **)
let extract v e b =
  take (drop v b) (sub e b)

I was pretty happy, so I tried some more. Each time, using nth, I managed to write a precise post condition and to prove it holds true. For instance, given map to apply a function f to each element of a vector v:

#[program]
Fixpoint map {a b n} (v : vector a n) (f : a -> b)
  : { v': vector b n | forall i,
        nth v' i = option_map f (nth v i) } :=
  match v with
  | vnil => vnil
  | vcons a v => vcons (f a) (map v f)
  end.

Next Obligation.
  induction i.
  + reflexivity.
  + apply e.
Defined.

I also managed to specify and write append:

#[program]
Fixpoint append {a n m}
    (v : vector a n) (u : vector a m)
  : { w : vector a (n + m) | forall i,
        (i < n -> nth w i = nth v i) /\
        (n <= i -> nth w i = nth u (i - n))
    } :=
  match v with
  | vnil => u
  | vcons a v => vcons a (append v u)
  end.

Next Obligation.
  split.
  + now intro.
  + intros _.
    now rewrite PeanoNat.Nat.sub_0_r.
Defined.

Next Obligation.
  rename wildcard' into n.
  destruct (Compare_dec.lt_dec i (S n)); split.
  + intros _.
    destruct i.
    ++ reflexivity.
    ++ cbn.
       specialize (a1 i).
       destruct a1 as [a1 _].
       apply a1.
       auto with arith.
  + intros false.
    lia.
  + now intros.
  + intros ord.
    destruct i.
    ++ lia.
    ++ cbn.
       specialize (a1 i).
       destruct a1 as [_ a1].
       apply a1.
       auto with arith.
Defined.

Finally, I tried to implement map2 that takes a vector of a, a vector of b (both of the same size) and a function f : a -> b -> c and returns a vector of c.

First, we need to provide a precise specification for map2. To do that, we introduce option_app, a function that Haskellers know all to well as being part of the Applicative type class.

Definition option_app {a b}
    (opf: option (a -> b))
    (opx: option a)
  : option b :=
  match opf, opx with
  | Some f, Some x => Some (f x)
  | _, _ => None
end.

We thereafter use <$> as an infix operator for option_map and <*> as an infix operator for option_app. *)

Infix "<$>" := option_map (at level 50).
Infix "<*>" := option_app (at level 55).

Given two vectors v and u of the same size and a function f, and given w the result computed by map2, then we can propose the following specification for map2:

forall (i : nat), nth w i = f <$> nth v i <*> nth u i

This reads as follows: the ith element of w is the result of applying the ith elements of v and u to f.

It turns out implementing map2 with the Program framework has proven to be harder than I originally expected. My initial attempt was the following:

#[program]
Fixpoint map2 {a b c n}
    (v : vector a n) (u : vector b n)
    (f : a -> b -> c) {struct v}
  : { w: vector c n | forall i,
        nth w i = f <$> nth v i <*> nth u i
    } :=
  match v, u with
  | vcons x rst, vcons x' rst' =>
      vcons (f x x') (map2 rst rst' f)
  | vnil, vnil => vnil
  | _, _ => !
  end.

Illegal application:
The term "@eq" of type "forall A : Type, A -> A -> Prop"
cannot be applied to the terms
 "nat" : "Set"
 "S wildcard'" : "nat"
 "b" : "Type"
The 3rd term has type "Type" which should be coercible
to "nat".

So I had to fallback to defining the function in pure Ltac.

#[program]
Fixpoint map2 {a b c n}
    (v : vector a n) (u : vector b n)
    (f : a -> b -> c) {struct v}
  : { w: vector c n | forall i,
        nth w i = f <$> nth v i <*> nth u i
    } := _.

Next Obligation.
  dependent induction v; dependent induction u.
  + remember (IHv u f) as u'.
    inversion u'.
    refine (exist _ (vcons (f a0 a1) x) _).
    intros i.
    induction i.
    * reflexivity.
    * apply (H i).
  + refine (exist _ vnil _).
    reflexivity.
Qed.

Is It Usable?

This post mostly gives the "happy ends" for each function. I think I tried too hard for what I got in return and therefore I am convinced Program is not ready (at least for a dependent type, I cannot tell for the rest). For instance, I found at least one bug in Program logic (I still have to report it). Have a look at the following code:

#[program]
Fixpoint map2 {a b c n}
     (u : vector a n) (v : vector b n)
     (f : a -> b -> c) {struct v}
  : vector c n :=
  match u with
  | _ => vnil
  end.

It gives the following error:

Error: Illegal application:
The term "@eq" of type "forall A : Type, A -> A -> Prop"
cannot be applied to the terms
 "nat" : "Set"
 "0" : "nat"
 "wildcard'" : "vector A n'"
The 3rd term has type "vector A n'" which should be
coercible to "nat".

Implementing Strongly-Specified Functions with the refine Tactic

January 11, 2015

Implementing Strongly-Specified Functions with the `refine` Tactic

coq

I started to play with Coq, the interactive theorem prover developed by Inria, a few weeks ago. It is a very powerful tool, yet hard to master. Fortunately, there are some very good readings if you want to learn (I recommend the Coq'Art). This article is not one of them.

In this article, we will see how to implement strongly specified list manipulation functions in Coq. Strong specifications are used to ensure some properties on functions' arguments and return value. It makes Coq type system very expressive. Thus, it is possible to specify in the type of the function pop that the return value is the list passed as an argument in which the first element has been removed, for example.

Is This List Empty?

It's the first question to deal with when manipulating lists. There are some functions that require their arguments not to be empty. It's the case for the pop function, for instance it is not possible to remove the first element of a list that does not have any elements in the first place.

When one wants to answer such a question as “Is this list empty?”, he has to keep in mind that there are two ways to do it: by a predicate or by a boolean function. Indeed, Prop and bool are two different worlds that do not mix easily. One solution is to write two definitions and to prove their equivalence. That is forall args, predicate args <-> bool_function args = true.

Another solution is to use the sumbool type as middlemen. The scheme is the following:

Defining predicate : args → Prop
Defining predicate_dec : args -> { predicate args } + { ~predicate args }
Defining predicate_b:

Definition predicate_b (args) :=
  if predicate_dec args then true else false.

Defining the `empty` Predicate

A list is empty if it is [] (nil). It's as simple as that!

Definition empty {a} (l : list a) : Prop := l = [].

Defining a decidable version of `empty`

A decidable version of empty is a function which takes a list l as its argument and returns either a proof that l is empty, or a proof that l is not empty. This is encoded in the Coq standard library with the sumbool type, and is written as follows: { empty l } + { ~ empty l }.

Definition empty_dec {a} (l : list a)
  : { empty l } + { ~ empty l }.
Proof.
  refine (match l with
          | [] => left _ _
          | _ => right _ _
          end);
    unfold empty; trivial.
  unfold not; intro H; discriminate H.
Defined.

In this example, I decided to use the refine tactic which is convenient when we manipulate the Set and Prop sorts at the same time.

Defining `empty_b`

With empty_dec, we can define empty_b.

Definition empty_b {a} (l : list a) : bool :=
  if empty_dec l then true else false.

Let's try to extract empty_b:

type bool =
| True
| False

type sumbool =
| Left
| Right

type 'a list =
| Nil
| Cons of 'a * 'a list

(** val empty_dec : 'a1 list -> sumbool **)

let empty_dec = function
| Nil -> Left
| Cons (a, l0) -> Right

(** val empty_b : 'a1 list -> bool **)

let empty_b l =
  match empty_dec l with
  | Left -> True
  | Right -> False

In addition to 'a list, Coq has created the sumbool and bool types and empty_b is basically a translation from the former to the latter. We could have stopped with empty_dec, but Left and Right are less readable that True and False. Note that it is possible to configure the Extraction mechanism to use primitive OCaml types instead, but this is out of the scope of this article.

Defining Some Utility Functions

Defining `pop`

There are several ways to write a function that removes the first element of a list. One is to return nil if the given list was already empty:

Definition pop {a} ( l :list a) :=
  match l with
  | _ :: l => l
  | [] => []
  end.

But it's not really satisfying. A pop call over an empty list should not be possible. It can be done by adding an argument to pop: the proof that the list is not empty.

Definition pop {a} (l : list a) (h : ~ empty l)
  : list a.

There are, as usual when it comes to lists, two cases to consider.

l = x :: rst, and therefore pop (x :: rst) h is rst
l = [], which is not possible since we know l is not empty.

The challenge is to convince Coq that our reasoning is correct. There are, again, several approaches to achieve that. We can, for instance, use the refine tactic again, but this time we need to know a small trick to succeed as using a “regular” match will not work.

From the following goal:

  a : Type
  l : list a
  h : ~ empty l
  ============================
  list a

Using the refine tactic naively, for instance, this way:

  refine (match l with
          | _ :: rst => rst
          | [] => _
          end).

leaves us the following goal to prove:

  a : Type
  l : list a
  h : ~ empty l
  ============================
  list a

Nothing has changed! Well, not exactly. See, refine has taken our incomplete Gallina term, found a hole, done some type-checking, found that the type of the missing piece of our implementation is list a and therefore has generated a new goal of this type. What refine has not done, however, is remembering that we are in the case where l = [].

We need to generate a goal from a hole wherein this information is available. It is possible to use a long form of match. The general approach is this: rather than returning a value of type list a, our match will return a function of type l = ?l' -> list a, where ?l is a value of l for a given case (that is, either x :: rst or []). Of course and as a consequence, the type of the match in now a function which awaits a proof to return the expected result. Fortunately, this proof is trivial: it is eq_refl.

  refine (match l as l'
                return l = l' -> list a
          with
          | _ :: rst => fun _ => rst
          | [] => fun equ => _
          end eq_refl).

For us to conclude the proof, this is way better.

  a : Type
  l : list a
  h : ~ empty l
  equ : l = []
  ============================
  list a

We conclude the proof, and therefore the definition of pop.

  rewrite equ in h.
  exfalso.
  now apply h.
Defined.

It's better and yet it can still be improved. Indeed, according to its type, pop returns “some list.” As a matter of fact, pop returns “the same list without its first argument.” It is possible to write such precise definition thanks to sigma types, defined as:

Inductive sig (A : Type) (P : A -> Prop) : Type :=
  exist : forall (x : A), P x -> sig P.

Rather than sig A p, sigma-types can be written using the notation { a | P }. They express subsets, and can be used to constraint arguments and results of functions.

We finally propose a strongly specified definition of pop.

Definition pop {a} (l : list a | ~ empty l)
  : { l' | exists a, proj1_sig l = cons a l' }.

If you think the previous use of match term was ugly, brace yourselves.

  refine (match proj1_sig l as l'
                return proj1_sig l = l'
                       -> { l' | exists a, proj1_sig l = cons a l' }
          with
          | [] => fun equ => _
          | (_ :: rst) => fun equ => exist _ rst _
          end eq_refl).

This leaves us two goals to tackle.

First, we need to discard the case where l is the empty list.

  a : Type
  l : {l : list a | ~ empty l}
  equ : proj1_sig l = []
  ============================
  {l' : list a | exists a0 : a, proj1_sig l = a0 :: l'}

  + destruct l as [l nempty]; cbn in *.
    rewrite equ in nempty.
    exfalso.
    now apply nempty.

Then, we need to prove that the result we provide (rst) when the list is not empty is correct with respect to the specification of pop.

  a : Type
  l : {l : list a | ~ empty l}
  a0 : a
  rst : list a
  equ : proj1_sig l = a0 :: rst
  ============================
  exists a1 : a, proj1_sig l = a1 :: rst

  + destruct l as [l nempty]; cbn in *.
    rewrite equ.
    now exists a0.
Defined.

Let's have a look at the extracted code:

(** val pop : 'a1 list -> 'a1 list **)

let pop = function
| Nil -> assert false (* absurd case *)
| Cons (a, l0) -> l0

If one tries to call pop nil, the assert ensures the call fails. Extra information given by the sigma type has been stripped away. It can be confusing, and in practice it means that, we you rely on the extraction mechanism to provide a certified OCaml module, you cannot expose strongly specified functions in its public interface because nothing in the OCaml type system will prevent a misuse which will in practice leads to an assert false. *)

Defining `push`

It is possible to specify push the same way pop has been. The only difference is push accepts lists with no restriction at all. Thus, its definition is a simpler, and we can write it without refine.

Definition push {a} (l : list a) (x : a)
  : { l' | l' = x :: l } :=
  exist _ (x :: l) eq_refl.

And the extracted code is just as straightforward.

let push l a =
  Cons (a, l)

Defining `head`

Same as pop and push, it is possible to add extra information in the type of head, namely the returned value of head is indeed the first value of l.

Definition head {a} (l : list a | ~ empty l)
  : { x | exists r, proj1_sig l = x :: r }.

It's not a surprise its definition is very close to pop.

  refine (match proj1_sig l as l'
                return proj1_sig l = l' -> _
          with
          | [] => fun equ => _
          | x :: _ => fun equ => exist _ x _
          end eq_refl).

The proof is also very similar, and are left to read as an exercise for passionate readers.

  + destruct l as [l falso]; cbn in *.
    rewrite equ in falso.
    exfalso.
    now apply falso.
  + exists l0.
    now rewrite equ.
Defined.

Finally, the extracted code is as straightforward as it can get.

let head = function
| Nil -> assert false (* absurd case *)
| Cons (a, l0) -> a

Conclusion

Writing strongly specified functions allows for reasoning about the result correctness while computing it. This can help in practice. However, writing these functions with the refine tactic does not enable a very idiomatic Coq code.

To improve the situation, the Program framework distributed with the Coq standard library helps, but it is better to understand what Program achieves under its hood, which is basically what we have done in this article.

Thomas Letan's Blog

What happened in February 2026?

What happened in February 2026?

Yet Another Claude Code Wrapper

I Built Myself a Cloudlab

Teaching Firefox About slack:// URLs

Teaching Firefox About slack:// URLs

What Happened in January 2026?

What Happened in January 2026?

tinkerbell’s First Month

Meeting with Claude Code

New RSS Feeds

How I Want to Use LLMs in 2026

How I Want to Use LLMs in 2026

I Cannot SSH Into My Server Anymore (And That’s Fine)

I Cannot SSH Into My Server Anymore (And That’s Fine)

Container-Centric, Declarative, and Low-Maintenance

Assembling tinkerbell

Container-Centric, …

Declarative, …

The Proof of Concept

The MVP

And Low-Maintenance

The Road Ahead

Peer-Programming in Modern OCaml with ChatGPT and Gemini

Peer-Programming in Modern OCaml with ChatGPT and Gemini

TL;DR

Editor Integration

Speech Recognition with Vosk

Prompting Local LLMs with Ollama

How It Started

More LLMs Lies

The Questionable Side Quest of Implementing a Connection Pool for cohttp-eio

Wrapping-up a Minimal Ollama Chat

Putting Everything Together

Final Words

What Happened in 2024?

What Happened in 2024?

Free and Open Source Software

Blog posts

$WORK

Talks

Sport

Final Notes

Serving This Article from RAM for Fun and No Real Benefit

Serving This Article from RAM for Fun and No Real Benefit

Embedding My Website in a Binary

Serving the content with Dream

Naive Approach

Handling index.html Synonyms

Supporting Content-Type

Compressing if Requested

Caching

Building and Deploying the Website

Conclusion

On Vestigial Structures

On Vestigial Structures

Introducing bepo-tsrn.nvim

Introducing bepo-tsrn.nvim

Installing a LUKS-Encrypted Arch Linux on a Vultr VPS

Installing a LUKS-Encrypted Arch Linux on a Vultr VPS

Booting on the Installation ISO

Remote Access to jasmine

Disk Partitions

Filesystems

/boot Partition

LUKS-Encrypted Root Partition

Installing Arch Linux

sshd

TinySSH

Grub

Networking

Conclusion

Using git maintenance with Encrypted SSH Keys

Using git maintenance with Encrypted SSH Keys

Building Static Binaries for OCaml Projects

Building Static Binaries for OCaml Projects

Setting-Up a Dedicated dune Profile

Setting Up a Specific Opam Switch

Verifying the Resulting Binaries

Teaching Firefox About `slack://` URLs

`tinkerbell`’s First Month

Assembling `tinkerbell`

The Questionable Side Quest of Implementing a Connection Pool for `cohttp-eio`

`$WORK`

Handling `index.html` Synonyms

Supporting `Content-Type`

Introducing `bepo-tsrn.nvim`

Remote Access to `jasmine`

`/boot` Partition

`sshd`

Using `git maintenance` with Encrypted SSH Keys

Setting-Up a Dedicated `dune` Profile

Extensible Records in OCaml Using `dmap`

How I Keep Using Stacked Git at `$WORK`

How I Use Stacked Git at `$WORK`

Implementing an Echo Server in Coq with `coqffi.1.0.0`