Fortran vs Numpy vs Julia, and HDF5

My background is as follows:
* I used Fortran for my PhD. Typical programs were about 100 lines long (but they were difficult lines!), and haven’t touched it much since then
* Extensive Python experience, and I’ve recently tried to convert to “python 3 only”. Recent start with numpy, and all that matplotlib goodness * Just started learning Julia

I’ve recently been much more interested in financial data analytics – hence my exploration of all things numerical.

I have been using HDF5 as a container for my data. It, too, is something I’m new to. I was beguiled by the sales pitch that it is a good way to store data as compared to, say CSV (which is how my data is captured). To be honest, my CSV files are small, so using the HDF5 file format to store the data is probably akin to using a sledgehammer to crack a nut. So there’s a fair bit of learning involved in working out how to use HDF5. I’m using it as a “fun” learning exercise, but if I were to analyse cost versus benefits dispationately, I would likely conclude that HDF5 wasn’t worth the effort; for my case, at least. I’ve found that HDF5 does have a decent benefit, though: it is a standardised format. The CSV files I obtain have all sorts of irregularities to them. In fact, it seems that the sources I have obtained them from have tried every permutation they can think of for subverting the CSV file format. At least when the data has been massaged into HDF5, there’s no room for confusion as to how the data should be accessed.

I found the Python scientific libraries to have a big learning curve. It is typical that many of the libraries don’t work in the way that one expects them to conceptually. This is not a slight on the Python libs. It just seems the way of the world that in programming, a certain alignment of stars is required to persuade the computer to behave in a way that you want.

Julia is turning out to be an interesting language, and I like the way that dealing with arrays is usually more concise than with Python. Benchmarks tend to show Julia is faster than Python, but I think that’s relative. The problem tends to be when you load a module for the first time. Julia has to compile it. That tends to be a slow old process. I’m not the only one that has experienced this. I saw an online YouTube video where Julia was demonstrated in an IPython notebook. If you restart the kernel, then you are inevitably hanging around for some crucial library to recompile. The demonstrator was obviously a little frustrated by this. That is understandable. In fact, I’m pretty sure that it will hurt Julia’s adoption. I am aware that one can build a Julia executable with your favourite modules compiled in, but it’s not an ideal solution.

It’s not an unfixable problem. Hopefully Julia will at some point in the future create bytecode, in the same way that Python does. Then Julia should feel much faster than Python.

HDF5 was quite tedious to use under Fortran, although I am sure I will become more comfortable with it as time progresses.

But here’s the thing. Whilst there is some more bookkeeping under Fortran, I’m not actually sure that Python and Julia bring much to the table in being able to reduce code size. After all, Fortran does provide array arithmetic. Sometimes the complexity of the algorithm means that a supposedly more expressive programming language wont reduce the line count.

Food for thought. Perhaps Fortran does suffice after all. It will certainly run quicker.

Posted in Uncategorized | Leave a comment

The Footsie versus the midcaps and AIM

Abstract In this notebook, I take a look at the relationships between the returns on the Footsie, and those for the midcaps and the AIM over the last decade. I found that there is a strong correlation between the Footsie and the midcaps, but a weaker correlation to the AIM. The midcaps have tended to amplify the gains of the Footsie, but the AIM market has lost money over the last decade.

http://pdf.markcarter.me.uk/indices2014-12-15.html

For the very geeky out there, I have bundled the underlying ipython3 notebook and data in a zip file: http://pdf.markcarter.me.uk/indices2014-12-15.zip, so you can experiment with it if you wish.

Posted in Uncategorized | Tagged , | Leave a comment

Confirmation of bear market?

The Footsie has been very predictable this year: buy when it is oversold, and sell when it is overbought. It was last overbought at the end of November 2014, at around 6730.  Since then, it has come off the boil, and is now 6600. There’s a more worrying trend:

bitmap

I have marked “highs” with red circles, and “lows” with blue circles. The market appears to now be selling off again. Notice how the second high is lower. We are now entering an interesting phase: will the lows get lower? I suspect it may take a couple of months to answer that question. But, if the low is lower, then I would take that as confirmation that a bear market had been in effect since around May this year. Odds would likely favour quality growth companies on reasonable valuations as holding up best if this turns out to be the case.

Valuations don’t look particularly stretched, but you may need to hold onto your hat anyway.

Just my 2 cents.

Incidentally, you may be interested to know that the chart above was prepared using IPython and matplotlib. The cirecles were added in using Inkscape. I’m becoming increasingly impressed with using IPython and the SciPy suite of modules for performing exploratory data analysis. I toyed with the idea of getting Mathematica, but that costs money, and there is so much that can be done analytically using Python. I would be interested in hearing if anybody thought that Mathematica was the only way to go.

Update 10-Dec-2014: The 200dMA is currently in downtrend:

index

Detractors to the market top counter with the argument that the S&P 500 is making all-time highs.

Note, however, that the S&P 500 is up 12.1% YTD, whilst the UKX (Footsie is down 2.7% YTD. So there’s a difference in fortunes. Oil and commodities are being clobbered at the moment, and they form a large part of the Footsie. BP has gone nowhere for years and is heading back down towards its 2010 Deepwater Horizon crisis levels. It is also laying off staff on the back of poor oil prices. Some oil services companies are taking a high hit: PFC (Petrofac) is down 40% YTD. ITRK (intertek), which has an oil services division, is down 30% YTD. The weakness in oil is not wholly to blame, though. ITRK was on a lofty valuation, and its growth has moderated, leading to a significant re-rating.

There’s a debate as to whether depressed oil prices is bullish or bearish. Ken Fisher covered this topic in his book “The Only Three Questions That Count”, and concluded that rising oil prices was bullish. Rising prices was a sign of economic health.

Stockopedia’s technicals are worth a look too: there are 1311 stocks below their 200d MA, significantly more than the 755 stocks above their 200dMA. That statistic should be interpreted as a sign of bearishness.

Caveat: this blog post should be considered “just a bit of fun”. I myself am significantly invested in stocks. So it’s a “prediction” not to be taken seriously.

Posted in Uncategorized | Leave a comment

Momentum experiment

Work by me looking at investing in the top momentum performers over 6 months.

It is an html rendered document from an ipython3 workbook. If anyone wants the original workbbok, then let me know, and I´ll make the link available. That way, you can play with the data yourself.

http://pdf.markcarter.me.uk/momo2014-11-26.html

Posted in Uncategorized | 2 Comments

My drawing for November

I have departed from Manga for November, and decided to make a sketch of Walter White, below.

white

I’m a beginning artist, so be gentle with me.

I have noticed that I receive far more comments and appreciation for my artwork than I do for my investing posts, despite this blog being primarily about investing. I wonder what that says about the psychology of artists versus investors and programmers.

Both artists and programmers tend to be both creative and introverted. Perhaps one significant difference is that artists relate to work by how they “feel” about it.

I would be interested to hear others’ thoughts on why they think that artists seem more communicative generally.

Posted in Art | Tagged , , , | Leave a comment

Cubietruck reactivated, gets a new home.

My internet is proving very unreliable, which hardly makes it ideal as an email server.

I decided to ditch hosting my own email server, and bought a dedicated one from Easily. It has a web interface if I choose to use it, but I prefer to use Thunderbird. I’m glad that I decided to go down this route, as it only costs me £10pa, and I can always go back to self-hosting if I decide that I don’t want to spend the money. I figure that it is worth it for the guaranteed uptime, though.

My LAN is acting a bit funny ever since we had the lightning storm. I have eliminated  damage to the internal computers, but I’m not 100% sure on the router itself. I’m thinking of getting a network switch to see if internal routing improves. I half-suspect that ‘tmux’ is causing problems with network resets, as unlikely as that may seem to an outsider.

Earlier this year, my cubietruck was taken out of commission because I bought a much-improved server. I have decided to reinstate it for testing purposes. I am using it as a backup ssh server. I have also decided to have it hosting my website. I am more sanguine about my website going down than my email. I’m also thinking about using my cubietruck as a NAS.

Given that my cubietruck is now likely to see plenty more service, I decided it was time to house it in a whole new lunchbox:

thumb

Lunchbox 2.0 is slightly smaller. When I was “customising” the original lunchbox, I had to put a couple of holes it it. The first hole I made was by driving a nail through the side. Bad idea! Plastic is brittle, so it created some fractures. Opening and closing the lunchbox for access just worsens the fracturing.

In my new design, I used a heated nail to melt an initial hole in the plastic. I then used a file to increase the size of the holes, as required. I also abandoned the idea of cutting a groove from the open in of the box. The idea is that by cutting a narrow slot, the power lead could fit into the slot, but the barrel on the lead would prevent it from being pulled out.

I had a better idea. The holes are now wide enough to accommodate the full width of the lead, and are entirely within the body of the box. This gives the box greater strength. In order to stop leads being jerked out, I used plastic clothepins.

The whole thing looks it was inspired from Orac out of Blakes Seven, and I would like to create a more elegant solution. That would likely be too complicated, though.

The image that I have shown above was created using IPython3 notebook from Anaconda and the pillow image processing library. I’m finding IPython great for experimental work, and I have been using it to investigate some momentum investing. I have some results, but I haven’t polished it off. The results are interesting.

Posted in Computers | Tagged , , , | Leave a comment

Article: Fund manager performance

Abstract: In this notebook, we seek to address the problem: does an FM (Fund Manager) or investor beat the index? The problem is looked at from two angles: a frequentist hypothesis test using the binomial distribution, and a Bayesian statistician using a Beta prior.

Link: http://nbviewer.ipython.org/url/pdf.markcarter.me.uk/bayfund.ipynb

This has been an interesting experiment for me. I was interested in writing about testing whether a fund manager could outperform an index. I would normally have written in LaTex, but I decided to experiment with iPython3. iPython3 is a great little bolt-on for Python that all the brainy boys are using these days. iPython3 is often combined with scipy for scientific computing. You end up with a cross between Mathematica, where you can do exploratory computing; and Knuth’s Literate Programming, where you intersperse code with text. It means that you don’t just throw code at the reader, but walk them through an argument in a discursive style.

The way everything is set up might seem a little strange. The file is in the form of a python notebook (.ipynb), that can actually be downloaded from my website:
http://pdf.markcarter.me.uk/bayfund.ipynb
That wont necessarily do you any good, because it is actually a JSON file used by iPython3. If you have iPython3 set up, then you can experiment with it yourself. Otherwise, there is a site called nbviewer.ipython.org that actually renders the notebook for you, so that you can view it as an html page. It gives you very nice rendering, and you can even see the graphical output. I’m quite impressed. I dare say it is straighforward to render the notebook to html itself, but I haven’t gone down that route yet.

Posted in Uncategorized | Leave a comment