Beginnings of a rewrite in #Forth

I my previous post, I said that I had written a little app in COBOL, and was interested in re-writing it in Forth. I am using gforth, which is an ANS Forth.

The problem I am first trying to tackle is converting numeric text into numbers. So, given s” 1963″, I should be able to put the number 1963 on the stack. Here is my attempt so far:

: 'space 32 ;
: space0 dup 'space = if drop '0 then ;
: >digit space0 '0 - ;

create str
: str++ str @ 1+ str ! ;
: sd str @ c@ >digit  + str++ ;
: sd10 sd 10 * ;
: i9999 0 sd10 sd10 sd10 sd ;

: set-str drop str ! ;

I have the variable str to point to the current position in text. str++ moves that pointer along. space0 converts spaces to ASCII ‘0, which is used by >digit to convert ASCII characters to numbers. It assumes that each ASCII character is a digit. I do not check that.

So, to convert text to an integer, you take an ASCII characters, convert then to numbers, and add it to the accumulated valued multiplied by 10.

Upon reflection, the above code could probably be better rewritten as

: sd 10 * sd ;

meaning that i9999 can be simplified to

: i9999 0 sd10 sd10 sd10 sd10 ;

The choice of the word i9999 might seem a bit puzzling at first, but it is meant to look a little a COBOL PIC statement. Maybe a word like PIC-9999 might have been better. Or perhaps just 9(4). The beauty of Forth is that you can actually define that to be a valid word, and it looks exactly like a COBOL PIC declarator.

It is now possible to parse a string like “1963” and produce the correct answer:

s" 1963" set-str i9999 . cr

Now that we have that in place, we can parse decimals as integer representations relatively easily:

: i9999.99 i9999 10 * str++ sd10 sd ;

Again, out word looks like a PIC statement.

So when we type

s" 1963.45" set-str i9999.99 . cr

we obtain the output 196345

It seems that Forth programming has an interesting property: you start in a deep hole, but you gain momentum quickly as you near the surface. I also note that one can perhaps be more generalised than in COBOL. For example, I don’t think there’s a neat way for COBOL to cope with input that has explicit decimal points in them. The way I solved the problem was to have separate integral and fractional variables, and meld them together using a computation. Hardly ideal! In Forth, I can just type str++ in my code to skip over where the decimal point is.

It’s interesting, and I’m going to take my experimentation with Forth further. There’s work involved that could be taken for granted in COBOL, but I guess that once you have everything in place, you can re-use it for other projects.

I suspect that a lot of people are put of Forth because you have to start from the atomic level. As the saying goes: you can do anything you like with Forth, but you have to do it yourself.

 

Posted in Uncategorized | Leave a comment

I wrote an app in #cobol

I started to write a little COBOL program last year, but quickly abandoned it. I decided to have another go at COBOL recently, and I have successfully completed my project.

I have made the program available as a gist. What it does is take a share portfolio of tickers (a company’s unique symbol) and portfolio values, cross-references them to a table of sectors, and rolls them up to create a table of portfolio value by sector and percentage of portfolio. That way, you can see if your portfolio is too over- or under- weighted in a particular sector.

It was a fun exercise, and took less than a week working less than an hour a day. The program was 227 lines long.

Data declarations amounted to around half the code, and included a table created at compile-time. I could have created a denser version in most languages, I feel.

COBOL was quite verbose, despite it being a so-called declarative language. I felt that I could probably have written a shorter version in C.

The compiler created a nice compact executable, weighing in at 21k on Linux. If I had written it in C I would have undoubtedly had an executable of the same order of magnitude. If I had written it in C++, I am sure the executable would have been much larger, as I would have used vectors and strings. When you start pulling those kinds of dependencies into the build, your executable tends to expand noticeably.

Abstraction seemed to be quite low in my COBOL code, and you have to do some bookkeeping. I felt that the code ought to be at a higher level. I’m pretty sure I could write at a higher level in a language such as C++.

An example where I found some repetitiveness was in file reading, such as in this section:

       read-epic-rep section.
           display "reading epic file"
           open input epics-file
           move 'N' to  eof
           perform until eof = 'Y'
                   read epics-file next record into epics-file-line
                           at end move 'Y' to  eof
                           not at end perform process-efl
                   end-read
           end-perform
           close epics-file
           . 

When I wanted to read another file, it’s basically a copy-and-paste job, with changes made to the calling routines. It would be nice to have a routine where I could say “put this file into that table”, and it would just do it. Maybe if I read about more modern standards of COBOL I would be able to see a way.

I had problems with compilation due to my complete unfamiliarity with the language. This should be expected, of course, and is not a bad reflection on COBOL per se. When I finished with the project, I felt I had a pretty good grasp of the syntax that I needed to use, and could write a program that needed to do what I wanted it to. I am sure I would be able to use many more refinements given more experience.

I’m still a bit confused about the distinction between sentences and paragraphs. I’m under the impression I’m not the only one. Periods (“.”) seem to demarcate both sentences and paragraphs. This is a problem because it crucially determines program flow! The trick seems to be to use less periods. Hmmm.

I thought that COBOL should have handled outputs a little more elegantly. COBOL’s internal representation is different from its output. So, for example, in order to calculate percentages for a category, I defined:

02 cat-percent pic 999v99 value is 0.

In order to print it out neatly, I had to define:

02 cat-percent-out pic ZZ9.99.

and then do a

move cat-percent to cat-percent-out

before displaying it in a more human-readable form.

I suspect that COBOL is very memory-efficient. I am assuming that the working storage section basically defines the whole memory layout of the data. I also assume that the procedure division doesn’t require a stack, as there are no functions, just sections, which could be implemented as jumps.

There’s two schools of thought about the verbosity of COBOL. Its purported advantage is that it makes it easy to read. There’s something of “silver bullet” trap there. The logic is that because it’s written in English, non-programmers can understand it.

The chances are, though, that

ADD X TO Y GIVING Z

is actually less readable than

Z = X + Y

And to be honest, if you have problems with the second version, you have no business reading programmes anyway.

The pro-verbose side also miss the real issue of inherent verses accidental complexity (or as I like to phrase it, intrinsic versus extrinsic complexity). Writing in English cannot remove intrinsic complexity: the problem is as complicated as it is.

So they miss this crucial point: understanding the semantics of a programme is relatively easy, whether it is in English or not, but understanding the structure of a programme is difficult.

I’ll grant you that a language is more suited to a particular task is likely to be more comprehensible than one that isn’t. However, you need to take care that you are not conflating different issues.

I am not advocating or denouncing COBOL per se on the matter, I am just noting that it may be a specious argument.

Would I program in COBOL again? Maybe, if it was a businessy-oriented type problem. It was certainly a fun exercise for me, as I have an interest in obscure and retro languages. No doubt COBOLers will shoot me down for the preceding sentence.

I could make my peace with COBOL. My own impression is that there is something higher-level waiting to get out, a kind of COBOL++. Or should that be ADD 1 TO COBOL?

It has made me curious as to whether writing some EDSL (Embedded Domain Specific Language) in C++ based on the principles of COBOL might be fruitful. You would then be able to do neat tricks like read entire files into tables as I had suggested before, specify relational constraints between tables that can be handled automatically, and even create automatic interfaces to SQL databases.

I have been thinking seriously about the EDSL approach. Before I do that, I think I will try to re-implement my app in Forth.

What I think is interesting is to take a design approach of COBOL as a way of doing it in Forth.

Interesting, no?

Comments, particularly in relation to my COBOL style? Is there a way that it could be more “idiomatic”? I enjoyed my little foray into COBOL, despite the somewhat overall negative tone of this post.

 

Posted in Uncategorized | 1 Comment

All languages can be made declarative

I was reading some interesting articles about simplifying programming:

  1. Out of the Tar Pit, on Reddit: https://redd.it/2ztw5o
  2. What, if Anything, Is a Declarative Language?, a blog article: https://is.gd/lBmORs

I’d like to add my own thoughts on “declarative” languages, which are often thought to be the Holy Grail of computing.

Declarative Languages are Walled Gardens. Wikipedia defines a walled garden as follows:

A closed platform, walled garden or closed ecosystem is a software system where the carrier or service provider has control over applications, content, and media, and restricts convenient access to non-approved applications or content. This is in contrast to an open platform, where consumers have unrestricted access to applications, content, and much more.

When I speak of walled gardens wrt declarative languages, I am not of course referring to vendor lock-in, but more in the way of “syntactic” lock-in. In other words, declarative languages do what the designers intended well, but present an impassible barrier when you want to do something the designers did not anticipate.

Take SQL, for example. SQL is a declarative language for querying databases. Simple queries are simple. But as the complexity of what you want to do increases, the queries become increasingly messy. You can’t assign results of queries to values, for example. So you can’t have intermediate query results. This means that all queries have to be done “in a oner”, potentially leading to some very hairy, convoluted code.

SQL doesn’t have state. So it’s difficult to do things like keeping a running balance – which is something pretty basic for an accounting package.

We already have the perfect tool for creating declarative languages: it’s called Lisp. If declarative languages were so great, why isn’t there an abundance of them in Lisp? Lisp macros allows you to extend the language in any way that you want. So it’s the perfect tool for declarative language construction. Except, we never seem to see big, stable, maintained, non-toy, declarative languages emerge from Lisp.

Maybe Lispers just aren’t interested in the same problems that the rest of us are. I dare say that many Lispers would state, rightly, that they are writing declarative languages. Technically this may be true, but I think they are languages “in the small”, designed to satisfy their own fairly esoteric (not meant as a pejorative) concerns. There’s been no big paradigm-busting declarative language to emerge from Lisp.

Wait, you don’t even need Lisp. Because really, every declarative language is, in essence, “just” an AST (abstract syntax tree). If you are too lazy to write your own parser, you can always hard-code your AST in the source language itself, and have it compiled in with the rest of your program. Lisp is not mandatory. It should be perfectly possible to do it in C, if a little messy. C++ would be nicer. Python might be even better, with Haskell at the pinnacle, perhaps. All you really need, though, is for the language to understand types and lists. In C++, for example, you could represent AST nodes as classes, and use vectors instead of lists.

Posted in Uncategorized | Leave a comment

Magic Hat – WIN stays in

The MHP (Magic Hat Portfolio) on Stockopedia (http://www.stockopedia.com/fantasy-funds/magic-hat-463/) is an experiment by me to see if a human can improve on a mechanical Greeblatt Magic Formula screen. I am trying to weed out “mistakes” that I feel the screening commits: unseasoned companies, scams, foreign companies (particularly Chinese), fishy accounting, and statistical quirks. Apart from that, I am agnostic as to the sector the company operates in, although I will try to avoid heavy concentration in any one sector. I will mostly apply “Strategic Ignorance”, by which I mean that I wont try to be clever in my stockpicking. My picking will be mostly mechanical. A summary of most of my Magic Hat articles can be found on the web page http://www.markcarter.me.uk/money/greenblatt.htm This will allow you to see, at a glance, what shares have been bought and sold in the past, as well as what shares have been rejected from consideration and why.

WIN (Wincanton) was due to be ejected from the portfolio. However, since it passes the Greenblatt Screen, it stays in.

Another easy month.

MHP is beating the FTSE350 on a 1-, 2-, 3- and 5- year basis, which is pleasing to see. The out-performance is not meteoric, but it’s nice to see that although the FTSE350 is down 2.7% over 3 years, the portfolio is up 20.1% over that period.

Posted in Uncategorized | Leave a comment

#cinelerra-cv: navigating the codec maze

Bottom line up front

Use DV files for importing and exporting to/from Cinelerra, transcoding with ffmpeg where necessary. MP3 files are good for sound-only tracks. Other formats are often problematical, and you’ll save yourself a lot of grief if you just stick with the tried and true.

The melange

Cinelerra has its own internal version of ffmpeg, meaning that both the standalone version of ffmpeg and vlc support a much wider range of containers and codecs.

Not all containers support all codecs. MP4 does not support the pcm_s16le audio codec, for example. pcm_s16le works perfectly in Cinelerra, and is used in DV files. But the fact that MP4 doesn’t support the codec means that you probably don’t want to work with MP4.

Cinelerra support of audio codecs seems patchy, making many containers/codecs unusable. What I have found is that sometimes the video will work fine, but I have problems with the sound, like it cutting out if I move the slider over Cinelerra’s viewer window.

The upshot is: just use DV, and Cinelerra should work smoothly.

DV is a space hog

An hour of DV will take up about 13GB – which is obviously a huge amount. Compare that with, say, a WTV (Window’s TV) file, which will use about 718MB.

DV is relatively quick to transcode, and is smooth for Cinelerra. So there’s that.

If I have a lot of footage at your disposal, but only want to use sparse sections of it, my current approach is to start with the footage in MOV format. I can select chunks of footage from the MOV file using ffmpeg, which I save in DV format. Here’s an example command:

ffmpeg -ss 01:02:30  -i reel01.mov -t 00:01:00 \
-vf scale=720x576 chunk01-01.dv

The ss flag specifies the start point for the chunk (1 hr, 2 min, 30 sec in my example), whilst the t flag  sets the duration of the clip (in this case 1 min).

MOV file

MOV files have the advantage over DV files in that they are much more compact, and have similar quality. They are slower to transcode, though.

Cinelerra seems to get “close” to importing MOV files.Often the sound will work if I play a clip from the beginning, but then run into problems if I want to start a clip at random points. MOV files that use the mp3 audio codec play in vlc flawlessly, but Cinelerra doesn’t like them.

Cinelerra’s rendering of MOV files are bad, and a no-go.

libvorbis is bad

I could not get either vlc or Cinelerra to play audio encoded in this codec.

Other ffmpeg audio codecs which work in vlc, but not in Cinelerra: ac3, libmp3lame

Audio codecs with were reported as unknown: libfaac, wav

An interesting audio codec is aac. This seems to be “the wave of the future”. It is designed to be a successor to MP3, and it used by YouTube, amongst others. Unfortunately, ffmpeg reported that aac support was experimental, requiring an over-ride flag to use it. I didn’t want to use an experimental feature, especially since the chances of Cinelerra supporting it were slim.

WTV format

WTV is an abbreviation for “Windows Recorded TV Show”. Vlc can play these files, but it stutters frequently. Cinelerra does not understand them at all.

Fortunately, you can use ffmpeg to convert them into a friendlier format. The quality will be just as good.

Posted in Computers | Tagged , | Leave a comment

Compiling #cinelerra-cv in Ubuntu Xenial

I was able to compile the latest version of cinelerra-cv from git:

git clone git://git.cinelerra-cv.org/CinelerraCV.git cinelerra-cv

I used commit 21049b7c from Apr 22 2016.

The following is neither a strictly necessary nor sufficient list of packages that need to be installed, but it should help you to get up to speed reasonably quickly from a new Xenial installation:

 

# standard'ish GNU tools
sudo apt-get install libtool # stops configure.ac AC_ENABLE_SHARED, ... probs

# mandatory formats 
sudo apt-get install \
libogg-dev \
libvorbis-dev \
libdv-dev \
libtheora-dev \
libjpeg-dev \
libfaac-dev \
libfaad-dev \
libtiff-dev \
libx264-dev \
libfftw3-dev \
libopenexr-dev \
uuid-dev \
libmjpegtools-dev \
liba52-dev \
libmp3lame-dev \
libsndfile-dev 

#optional formats
sudo apt-get install \
libraw1394-dev \
libiec61883-dev \
libavc1394-dev

# optional, but you might as well:
sudo apt-get install \
libqt4-opengl-dev \
libopencv-dev

 

Then you can build in a fairly standard way:

./autogen.sh
./configure
make
sudo make install

 

I have yet to test cinelerra in operation.

 

Have fun

Posted in Uncategorized | Tagged , , | 2 Comments

HOME and DEB followup

Six months ago I wrote about HOME (Home Retail) and DEB (Debenhams).

The story was that DEB had good price momentum, whilst HOME had poor momentum.HOME sunk on its trading update, whilst DEB gained. I said at the time that HOME looked value-trappy.

I had anticipated that HOME would continue to decline, whilst DEB would continue to improve. I was wrong, however. Here are their charts for the last 6 months:

home.png

 

 

deb.png

As you can see, HOME improved over the last 6 months, outperforming the index (FT100) by 56%, whilst DEB declined, under-performing it by 7%.

HOME’s shareprice has been helped by the takeover offer from SBRY (Sainsbury), so it would be difficult to know how things would have panned out without this exceptional event. Having said that, HOME’s cash pile made it more attractive as a bid target.

Momentum can reverse.

HOME 171p . DEB 79p. ASX 3426

Posted in Uncategorized | Leave a comment