#rakulang grammar hints about creating a templating system

Raku’s grammar system is very powerful, but often mystifying and frustrating. Some worked examples go a long way. This post isn’t a “how to create a templating system”, it just contains a few hints on getting started.

It turns out that a simple system is easy. I have in mind a way of extending my assembler written in Raku with a very naive substitution system. This will enable me to write assembler at a higher-level. Really, it will land me a third of the way towards BASIC.

Where might I be going with this? Well, instead of specifying a grammar for BASIC as is normally the case, I had an interesting idea of creating a kind of extension mechanism inspired by Forth. So you have a kind of BASIC, but with a plug-in extensible grammar system.

I’ll describe how such a curious feat might be achieved in a later post, assuming I decide to implement it. But I have digressed too far already. so, back to the topic at hand.

What I’m trying to achieve in the first instance is that you may have triggering syntax embedded inside general text, and you want to perform actions on this special text.

The example I chose to work with is wherever I see the word IF, I want to replace it by --GOT IF--, and where I see the word IFX, I replace it with ..GOT IFX.. . All other text should be left alone. So, pretty simple, but not necessarily easy to figure out how to achieve if, like me, you’re new to Raku’s grammars.

It turns out that the implementation (available as a gist) is not difficult at all:

grammar G {
	token TOP { <atom>* }
	token atom { 	IFX { print "\n.. GOT IFX..\n"; } 
			| IF {print "\n--GOT IF--\n"; }  
			| .  { print $/; }

my $str = "heIFllIFXo\n";


# outputs
# he
# --GOT IF--
# ll
# .. GOT IFX..
# o

There we go, simple as that. Some points to note: I have defined inputs as being made of “atoms”. An atom is either one of the keywords, or anything else. You can see token atom selects from a list of alternatives via the | operator.

Note also that the action that we perform is dependent on the actual match that succeeded. This facility may be obvious to some, but I didn’t know about it until a couple of days ago. I had always assumed that the action was based on the whole token. Like I said before, Raku grammars have a lot of stuff available, but it’s not necessarily obvious for the beginner. Trying to wade through a morass of documentation often doesn’t help. It is often tempting to ditch the whole approach and just hand-roll a grammar.

Note that IFX is define before IF, because if they were defined the other way around, then you could only match IF, and not IFX. For the same reason, the . , which matches anything, is defined last.

A final point to note is that $/ represents the “current match”. In the context of the code above, if we don’t match IF or IFX, we match something else, which we just echo out.

Great! Now go and implement the Unix tools m4 and groff in Raku. Whilst you’re at it, be sure to add programmability, a la awk.

That’s all for now. Stay safe. May you all be happy.

Posted in Uncategorized | 1 Comment

#rakulang An assembler for a virtual machine

I have implemented a Forth, a BASIC, and a formula interpreter for a spreadsheet. The idea of a virtual instruction set keeps cropping up from time-to-time with me.

In this post, I shall be looking into my github repo bennett . It contains source code from the book “Introduction to Compiling Techniques: A First Course using ANSI C, LEX and YACC”, by JP Bennett. I have made my own modifications to improve compilation on modern hardware, as well as re-implementing the assembler and disassembler in Raku (formerly Perl6, of course).

The purpose of the book is to implement a Basic-like language which supports recursion. The assembler is therefore a perfectly capable one. The assembly language consists of 15 instructions: HALT (to stop the machine), NOP (does nothing), TRAP (outputs a byte), ADD/SUB/MUL/DIV (arithmetic), STI/LDI/LDA/LDR (storing/retrieving values) and BZE/BNZ/BRA/BAL (branching).

The major limitations is that it does not take in keyboard input, and doesn’t have a facility for incorporating foreign functions. These shouldn’t be too difficult to add, especially the former.

The repo does have a C implementation of the VSL (Very Simple Language, i.e. a form of BASIC, which compiles to assembler), VAS (assembler, which compiles to machine instructions), VAM (a virtual machine executer).

Raku enthusiasts may be more interested in the stuff that I’ve implemented for Raku, though. It is available in the myvm directory . asm.p6 is the assembler, and disasm.p6 is the disassembler. It is satisfying to note that the assembler takes only slightly more than 100 lines to implement using Raku’s grammar facilities. The line count could be reduced further if offsets were stored in little-endian format rather than big-endian. I chose to stick with Bennett’s way of implementing things, though, for compatibility.

If you want to test things out, then compile the top level directory:

autoreconf -iv
cd examples

The examples will compile to VAS and VAM using Bennett’s implementations. You can write your own assembly programs, but you’ll probably want to use the examples in the first instance. If you are in the examples directory, and have already issued a make in accordance with the instructions above, then you can prove that the Raku assembler works by issuing a command like:

../asm.p6 fact.vas

This will create a file called out.vam, which should be identical to fact.vam. You can run the compiled code by issuing the command:

./vc out.vam

The next step might be to implement the virtual machine in Raku, and a little more ambitiously, the parser for VSL.

Some directions I personally might want to take:

  1. see if the assembly language is a good base implementation for a Schorre Meta-II compiler. That would be an interesting project.
  2. see if I can implement co-routines or continuations in an easy fashion.

If there’s significant interest, I’ll post some details on Bennett’s instruction set, and what’s going on under the hood of the compiler. The repo does contain some info in vam.txt for those interested in writing some assembler.

That’ll do for now, though.

Update 2020-06-30

I’ve made a few exciting updates to the assembler and VM. Surprisingly perhaps, Bennett’s VM didn’t have an N flag, to test for negativity. There were no branch instructions based on this. So you could only branch unconditionally, on zero, or not-zero.

Consequently, I have added BLTZ (branch on less than zero) and BGTZ (branch on greater than 0).

I have also added a SYS instruction, in order to make system calls (“interrupts”). The format for it is:


where n is the call number. Currently there is only one system call you can make: 0 (for getchar()). You can extend the VM to make other calls, though.

See examples/echo.vsl , which copies stdin to stdout. Here is its implementation:

\ added by mcarter 2020-06-30
\ test of syscalls
\ this just echos stdin to stdout

        SYS 0           \ getchar into R15
        LDR R15, R15    \ check for EOF
        BLTZ fin        \ if EOF then finished
        BRA loop

You can read more about the VM instructions here.

Posted in Uncategorized | 1 Comment

Father’s Day Card

Just a little watercolour card I made for Father’s Day on 21 June.


Posted in Uncategorized | Leave a comment

Ubuntu 20.04 MATE (Not a “review”)

I eagerly awaited the release of Ubuntu 20.04. Ubuntu 19.10 was buggy for me on a couple of machines. I seems likely that it’s something Canonical did wrong, rather than a specific hardware quirk. This forced me to install a previous version of Ubuntu. The only one I had available was 18.10.

18.10 worked OK, but the problem is that it stopped support, and I couldn’t upload the new software that I wanted. So I switched to Absolute Linux, which is derived from Slackware Current.

Although Absolute/Slackware is a nice distro, dependency management can be a chore. Sometimes the dependency “cascade” can be overwhelming; video editors, for example. I decided that the best way forward in some of those instances was to use AppImages.

I’d say that AppImages work fine half the time. For the other half, some kind of library needs to be installed in order to get them to work properly. Even then, I had some hard problems with QT5 or some GNOME dependencies.

I decided that the best thing to do was just stick to what worked, ignored what didn’t, and wait it out until Ubuntu arrived. I concluded that AppImages are worth a try as a method of last resort, but they are not a panacea. At the end of the day, dependency management is hard.

Absolute uses the Icewm WM. I like it, although it was a bit of a chore to smooth out some of the rough edges and tweak it how I wanted it.

Alas, it is becoming increasingly difficult to recommend Slackware given that it’s last stable release was in 2016. Whilst it’s true that “newer doesn’t necessarily better”, it’s also try that “newer doesn’t necessarily mean worse, either”. As software projects mature, they generally file down their flesh-ripping jagged edges.

My honest opinion is that the latest stable version of Slackware (14.2) is too old for typical desktop usage. Maybe it’s fine for server usage where you don’t want to install any fancy extra software. The standard response is “just use Slackware Current.” I think you’d be better off using Arch Linux in that case, or use a compromise like Absolute.

Ubuntu is a distro that I always return to. I opted for the MATE desktop version. GNOME is too bloated and unorthodox for my liking. I’m increasingly being able to tolerate GNOME, but it’s not my ideal choice, so why bother?

MATE is a good choice. I also like LXDE and Icewm. At a pinch, Fluxbox is a reasonable choice. I don’t really like Xfce. It always seemed buggy, ugly and finnicky to me.

At the moment, I’m flitting between using MATE and Icewm as my choice of WM. Icewm needs effort to make it do what I want it to do. Fortunately, I have pretty much done that through my tinkering with Absolute.

Icewm definitely has that 1995 aesthetic. You kinda get used to it, though, and even grow to prefer it. Fancier DE’s (Desktop Environments) can look “too” beautiful, if you can believe such a thing. Pretty, for sure, but in another sense, “overproduced.”

I have system restoration down to a fine art now, so I only have to clone my git repo and run a few installation scripts. That’s actually the neat thing about a distro like Ubuntu: I can just run a script that will install that I find crucial to my needs. I can’t do that with Slackware.

An LTS version of Ubuntu has come at an opportune time for me, too.  Windows 7 is acting all flakey now on my dad’s machine, so a system refresh is in order. So it’s good to be able to install a version of Ubuntu that has longevity.

I think it’s been over a year now since I last used Windows. Linux definitely makes for a better development environment, which is what’s important to me. Crucially, it covers internet browsing, email, Youtube and Skype. I’ve been using Skype more these days, and I’m glad Microsoft has a port. Thanks, Microsoft.

Zoom is something I’m also beginning to use and like now that COVID is here. I have used it to attend a “virtual pub” with my old university chums. It’s great, really, because we don’t all have to meet up in the same place. I’ve even been able to chat with my old professor who lives in Spain.

I have also attended a meeting with a discussion group in the US, and plan to continue that. So I’m really happy that Zoom is available on Linux.

It seems to me that Linux has all my bases covered now, so I hope never to hanker after the days of Windows.

Posted in Uncategorized | Leave a comment

#forthlang: a macro line trick

Bored with Perl? Looking for a cool language to learn? Then why not try Forth? This post is aimed to tease you as to the possibilities of Forth, and hopefully to get you to try it. As usual, I will use my own version of Forth, which you can find here.

I have an accounts package that I have written in Perl. I often think it’s fun to try to see how I might approach the problem if I wrote it in Forth. Perl processes a file containing a list of commands. Commands are either “#”, for comments, “ntran” for normal transactions, and “etran” for equity transactions.

In most languages, you’d have to write some kind of parser to break commands down into type and arguments, and dispatch on on the command. With Forth, you can use the Forth parser itself to do the parsing for you. You don’t even need to dispatch on the command; the commands can be defined as Forth words themselves, which the parser just executes.

Done well, Forth can therefore feel very much like a DSL. Although the plumbing might feel a bit messy sometimes, when you reach the top level, the code itself can look very “crisp”.

An “ntran” consists of a few fields: a date stamp, a debit account, a credit account, an amount, and a description. A typical example:

ntran 2020-04-18 bank cash 10.00 "banked some cash"


Let’s suppose we want to have variables for each of these fields. The standard way in Forth is to use the word VARIABLE, like so:

variable dstamp \ defines the variable for date-stamp
variable dr \ for the debit account

and so on.

But that gets a little tedious. Suppose we want to define multiple variables at once. To that end, it would be convenient if we had a word like VARS:, which scanned the rest of the input up until the newline, and defined new variables for each token on that line. In other words, we wanted to be able to do something like:

vars: dstamp dr cr amount desc

Here’s how we could implement it:

: VARS: begin parse-word dup while $create 0 , repeat drop ;

You may be able to see a begin … while … repeat loop there. It’s a loop that parses the next word in the input to a string (via parse-word), and then does something similar to the variable word (via $create 0 ,). The “drop” at the end clears a dangling parse-word.

We can then define the word “ntran” that reads word from the data, and assigns values to variables. We want to repeat the trick that we used for vars:, to write something like:

: ntran store: dstamp dr cr amount desc
     process-ntran ;

where PROCESS-NTRAN does some kind of further processing that needn’t concern us here. The word we’re interested in is STORE:, which will have the same outer looping construct over the line as VARS:, but a different inner body.

A pattern is developing, so let’s try to abstract away that pattern. We want to be able to write  something like:

: VARS:  line( $create 0 , )line ;
: STORE: line( swap ! )line ;

where LINE( is effectively the phrase “begin parse-word dup while”, and )LINE is the phrase “repeat drop”. We need to turn those “phrases” into “macros”. Here’s how:

: LINE( postpone begin postpone parse-word postpone dup postpone while ; immediate
: )LINE postpone repeat postpone drop ; immediate

In this case, the “macrofication” of the phrases is relatively straightforward. You define “macros” by making them immediate, and stick POSTPONE in front of each word. It’s not always that straight-forward, unfortunately, but in this instance we have avoided some of the complexities that can arise.

There are other possible strategies you could use to achieve the same effect, but I won’t explore them here.

I hope that you found that interesting and piqued your interest as to how Forth can be made to look like a very high level language with terse and flexible semantics, even if you didn’t understand much of it.

Update: I just realised that my code for STORE: won’t work as-is, as it needs to be IMMEDIATE. A correct implementation is:

: STORE: line(
postpone parse-word postpone pt find cell + postpone literal postpone ! 
)line ; immediate

We can see an implementation:

vars: a b c
: foo store: a b
a @ . b @ . cr

foo 10 20  \ outputs 10 20
foo 30 40  \ outputs 30 40



Posted in Forth | Leave a comment

Has the bell rung for the bottom?

I am now going to commit an act of utter insanity: by predicting that the market is near the bottom. To quantify that as a specific prediction: the FT100 is within 10% of its 2020 low, and will be higher in 6 months than it is now.

Why do I say that? An article was published on MarketWatch that what taken from Barron’s: “Fear Grips Markets Again. The Only Safe Place Is Cash”. I am not a subscriber, so I know next to nothing about its contents.

Another headline is also links: “The Latest Coronavirus Data Is Out. The Number of New Cases Worlwide Declined Today”.

In a post on Stockopedia, a user notes that South Korea, which has a similar population as the UK) has reported only 84 deaths after a month of the outbreak. A respondent to that post says that Italy has a population 10% smaller than the UK with 2500 deaths attributed to the virus.

Let’s see if I’m about to make an utter buffoon of myself. Follow-up in 6 months time.

FTSE100 5091


Posted in Uncategorized | 1 Comment

Magic Hat Portfolio: down 19% today

The markets are exhibiting extraordinary behaviour at the moment. The portfolio benchmark, the FT350 is down 8.5%. The MHP is down 18.5%, with everything at least down 10%, with the exception of FXPO, which is down “only” 7.3%.

Airline tour operator DTG (Dart) is down 42% after announcing that it has stopped flights to Spain. SLP (Sylvania Platinum) is down 36%.

The fall in the indices has been relentless. We’re down about 40% from the peak earlier this year.

The Dow circuit breaker kicked in earlier. I’m not sure the exact status right now, but the prognosis is bad.

The S&P500 is still on a PE of 17.2, which is still above the long-term average of around 15. Arguably, it still hasn’t factored in likely forward earnings despite the falls that it has suffered so far.

Could it fall a further 50% from here? Well, it’s not impossible, and there’s a good argument for it being within the realms of possibility. I’m not so bold as to suggest that it’s probable, though.

Stay safe out there.

Posted in Uncategorized | Leave a comment

Magic Hat Portfolio: a little update

After yesterday’s horrendous fall, the FT350 is now lower than it was when I first started the MHP (Magic Hat Portfolio) in February 2011. So nine years out, and the FT350 has returned a big fat zero. Less than zero, in fact.

A similar situation arose in 2016. The FT350 is not at its lowest point since then., though. The nadir happened in October 2011, where the FT350 has declined amost 20%.

Fortunately my stock selections themselves are in positive territory, having returned an annualised growth of 6%. That excludes dividends. A few months ago the growth rate was about 10%.

Let’s see what the future brings.

Stay safe out there.

Posted in Uncategorized | Leave a comment

TUI looks vulnerable to me

COVID is starting to ramp up in Europe now. Things are worsening in Spain, and it’s starting to look bad as a holiday destination. It’s difficult to gain a perspective in the UK, because I believe we are only at the thin edge of the wedge. It seems to be the same for the US. I do hope we avoid a situation like China and Italy. Realistically, though, I think the wheels have been set in motion, and it’s just a question of time.

If the US catches COVID bad then I expect stock markets to get a lot worse.

UK stock markets are down about 30%, so we’re in unequivocal bear territory. I do think markets will go lower. I’m actually a little optimistic about our eventual recovery from the virus, relatively speaking.

Anyway …

I note that tour operator TUI is down 14.3% to 398p. It does a lot of package holidays to Spain, so obviously it is going to be greatly impacted by the virus. One thing I did want to point out is just how vulnerable I think TUI is as a company.

A quick check on Stockopedia reveals that it has a market cap of £2.8b and an EV (enterprise value) of £7.9b. I view this as perilously high debt. TUI has operating margins of around 4%, and has plenty of fixed costs. It won’t take much to put this company into a loss, thereby increasing its debt load.

Put it this way: if I was offered an even-money bet as to whether it would survive or not over the next year, then I would put my money on “survive”. But make no mistake, TUI is vulnerable.

It’s a shame. I have taken many holidays through TUI. I found the company to be well-organised and the reps helpful. So I wouldn’t want it to see it go the way of Thomas Cook.

I did not take a holiday abroad last year on account of my passport near expiry and the existence of the tourist tax on the Balaerics. £75 to renew a passport is outrageous, too. I do resent being treated like a walking wallet.

Akthough I do miss going abroad a little, there’s something to be said for staying at home. If we get nice weather, then I sometimes wonder what the point of going abroad is. Having said that, I do live in Scotland! Brr.

Anyway, that’s enough nonsense from me for today. Stay safe out there.

Update 1: Someone mentioned SAGA, which is a travel operator for old crusties like me. I see it made a loss in 2019, so prospects for 2020 can’t ge good, either. SAGA is down 15% to 14.8p. It has a market cap of £193m, and an EV of £748. Verdict: too much debt, especially for such a cyclical sector. SAGA has a long trading history, but has only been publicly listed since 2014. I think it was a VC management buyout jobby. As you might expect for such a setup the company was pushed out to the market loaded with debt. It’s on a PE of 2.4 and a PBV of 0.2 and a momentum score of 15. Clearly Mr. Market thinks that the wheels are going to fall off the wagon on this one, and I wouldn’t necessarily question his judgement at this point.

Update 2: I didn’t want to post this update on Stockopedia because it might be considered too frivolous. The FT100 is now down 9.2%, similar to the Dow. A BB poster noted said that gold was falling, too, and that there was a simple explanation: margin calls. I found a video on YouTube from late 2018 entitled: “MARGIN CALL: Why The Next Market Crash Will Be Worse Than Anticipated.” Margin debt was at an all-time high when the video was made, and I don’t imagine things are any better now than they were a year-and-a-bit ago. The video may have been early, but may have been prescient. Make of it what you will.

Update 3: FT100 closed down 10.87%, making it the second-biggest fall on record. The index closed at 5237.48.

TUI 398p

SAGA 14.8p

Posted in Uncategorized | Leave a comment

My thoughts on the market

The number of cases of covid is currently small in the UK. The UK health minister accounced that she has it, and I surmise that it will now spread rapidly. We are experiencing a calm before the storm. I hope I am wrong, but I think that’s the way it’s going to go. I imagine it’s the same situation in the  US. If America goes, we’ll probably all go down.

As I noted elsewhere, the FT100 is in bear territory. My worry is, when markets decline by 20%, they don’t just decline 20%, they like to decline more. I don’t know by how much, but 30% seems reasonable. It could be more, it could be less.

I also noted that near-record 1-day declines haven’t appeared in isolation. So we’ve probably got one, two, or possibly more whoppers to come.

The RSI(14) of the FT100 is at 17.88, as it was a few days ago. Anything below 30 is considered oversold. So we are at levels that we could class as extremely oversold historically.

My speculation that we would have a sharp rebound has been wrong so far.

The declines that we have seen recently seem uncharacteristic of a typical end-of-bull market. My expected behaviour is that bull markets end with a whimper, not with a bang. The declines seem too sharp. It’s the end of bear markets that one normally associates with steep declines.

The fallout from covid has not yet filtered into company announcements, so there’s that to look forward to! According to Stockopedia, the FT100 is on a PE of 14. This is lower than average, but still in what i would consider an average range.

So I think there’s plenty of scope for further falls, especially given the fact that the UK has yet to see covid bite into the UK population and company results.

Stay safe out there.


Posted in Uncategorized | Leave a comment