Reading back over my preceding posts on this topic, it's fair to say I've been fumbling around quite a bit since first deciding to take on my project of building a language server for OPL, which makes today's post a rather more triumphant one because I can now say “I've made a thing”! Yes, my proof of concept language server is now up and running, and while it is not what you might call especially feature rich (it hasn't really got any features at all) it does function and has given me the confidence that I am - finally - on the right track.

A little history on what I've been trying over the past few weeks (months…) to get me here. You may recall from my last post that I had arrived at the idea of using standard in and out (stdio) for the communication protocol of my language server. This led me back to Jeff Chupp's article on writing a language server in Bash as it provided a nice rudimentary template for creating a bare bones language server that worked over stdio. I took Jeff's code, installed Neovim and found that it did indeed work much as he had laid out (with a few minor tweaks to suit the latest version of Neovim). The nice thing, by the way, about Neovim is that you can fire up a language server with a single line of Lua code. This makes it an excellent test bed for language server development.

Well, the Bash language server was all well and good, but as Jeff says - and I agree! - you don't really want to write an entire language server in Bash. I was already set on doing so in Pascal, so my next step was to see if I could replicate the Bash version but in Pascal. This didn't take me too long, and because it worked over stdio using Pascal's basic readln and writeln functions, I was able to test the behaviour on the command line. So far, so good; until I then tried starting it up via Neovim. This is as simple as hitting esape and running the following comand after opening Neovim:

:lua vim.lsp.start {name = "SimpleLS",cmd = {"/path/to/lsp/SimpleLS"}, capabilities
= vim.lsp.protocol.make_client_capabilities(),}

Unfortunately, this is where things took a turn for the worse. Neovim started the SimpleLS program just fine, but then failed to make its initialisation handshake. By adding in a simple Pascal-based file log to track the incoming message from Neovim, I could see that while the first readln was reading the Content-Length header of the request, the second readln call wasn't bringing in the JSON content of the message. It was as if the language server was just sitting there, waiting. Neovim, meanwhile, having received no response to its request, decided that there wasn't a language server there to take its call, and hung up. The process carried on running in the background, as confirmed by running ps -A | grep SimpleLS but inside Neovim, running the :checkhealth vim.lsp command showed no language server connection had been made.

I wondered if the issue was a lack of a line feed character at the end of the request message, so to get over this I converted my second readln into an old school C-style read of each individual char in the incoming request from Neovim, using a regex on the Content-Length header (successfully read by the first readln call) to guide the number of chars to read for the full JSON-RPC payload. This actually worked, and I was able to see the full request from Neovim inclusive of the JSON portion in my file log. Not only that, but having successfully obtained the full JSON-RPC request, my language server dutifully created the response message and wrote this out to the log file and to standard out via writeln. Yet, once again, I hit a blocker: Neovim didn't seem to be receiving this response.

I tried and tried, even at one point using the char-based approach that had worked for the reading of the request, but no dice. I eventually ran a strace to track exactly what Neovim was seeing on its read buffer, and compared this to the Bash language server: and where the Bash response message showed up in Neovim's read buffer, my poor old Pascal one did not. It was as if neither the output of writeln or write were visible to Neovim, even though they showed up on the command line if I ran the language server there directly. I wondered briefly if my log file was confusing things, but after commenting all log file writes out of my code, the same behaviour continued. At one point I even coded a dumb program that just returned the response message via writeln without even reading the request from Neovim; but this, too, failed to work and there was no evidence of it in strace.

This is when I decided I needed to try another approach. After some trawling around on the Lazarus forums, I stumbled across the concept of Pascal pipes. These allow you to define both input and output streams in your code and read from or write out to them as you require. I coded a very simple demo program that just echoed out whatever came in, and then called this from Neovim as if it were a language server. While it, naturally, caused Neovim to throw a wobbly, the evidence was there that it could now see the output of my Pascal program!

I then hacked out all the readln, read and writeln code of my original proof of concept language server, replaced it with the pipes and - it still didn't work! But, importantly, this was because of bugs in my code that I was able to quickly sort out. If you're interested, the LSP states you have to use two sets of \r\n (carriage return and linefeed) characters after your initial Content-Length header when forming a response. I was doing this with literal \r\n characters, but I needed to use two sets of #13#10 in Pascal instead, and this - finally - got me what I wanted to see: a request, followed by a response, followed by another ‘request’ (which is just Neovim saying “I see you, language server”). Result!

(Well, I had to then refine my code further to stop it bombing out when it received this second not-request from Neovim, because that second 'request' lacks an id key in the JSON - presumably because it's an information-only message that doesn't require a response. My code, however, was parsing the JSON to look for the value of the id key, and when it couldn't find one, I got a nasty EObjectCheck exception. But once that was cleared up, all was right in the world).

And that's what finally got me to the screenshot at the top of this post: a working, proof of concept language server with simple auto-completion on a selection of OPL keywords. In Neovim you activate that, by the way, using CTRL+X CTRL+O after typing as much of the word as you want to type before looking for a possible match. If it finds only one match for the characters you've already typed, it just auto-completes the entire word, but if there are multiple options (like in my screenshot where I typed just the letter E before hitting CTRL+X CTRL+O) it gives you a nice pop-up menu of options. I understand there are more advanced configurations you can write for Neovim that enable pop-ups automatically, but that's for another day.

So, to next steps: I need to refactor my proof of concept language server, as the code is looking especially shonky right now. It's also a completely procedural single unit code block and could do with some refactoring, and possible OOP improvements. This will hopefully mean it will become something that's easier for me to extend and enhance as I make my way through the various methods supported by the LSP and implement them for OPL. Once I get to that point, I promise to upload it to GitHub so you can see what I'm doing. It's not fit for public consumption right now, however!

This really does feel like proper progress, as I at last know what to do next and thus have the makings of a fully fledged project. All I really need now is… time!