voidynullness

The Terrible Turnbull and LNP NBN legacy

2019-05-15T20:32:00+10:00

Hey remember Malcolm Turnbull? Remember that guy? Remember him? He was the Prime Minister until not too long ago, wasn’t he? He was the also guy largely responsible for screwing up the NBN.

Remember the NBN? That huge national infrastructure project that would have brought great benefits to the nation — if Malcolm and the LNP Coalition government hadn’t screwed it up.

Malcolm Turnbull might not be dead yet, but his political career is, and its ghost will haunt the NBN for years to come. The NBN as envisioned by Labor is dead, but what remains is a zombiefied mess that will terrorise anyone expecting decent broadband in this country.

As this election campaign grinds into its last week, and draws to a merciful end, the Coalition is asking Australian voters to re-elect the same old government with new leader in Scott Morrison. It might be instructive to take a little walk down memory lane and look back on the largely lamentable legacy of its previous leader. The one who destroyed the NBN — a fact that seems to have been largely forgotten.

While not necessarily surprising, I do feel it’s certainly dispiriting how little the NBN has been mentioned during this election campaign. It’s not a surprise because technology issues rarely make it into the mainstream political discourse — there just aren’t enough votes there to matter, it seems. And computer nerds aren’t very good at organising collectively in a way that makes the powers-that-be take notice.

The NBN, despite being a relatively dull tech issue at its core, had a moment of mainstream political fame years ago, because of the costs involved, and because the Coalition sensed there was some cheap political points to be scored there — at the expense of the nation.

I’ve been trying to avoid writing (or thinking) about the NBN, because what’s the point anyway? Most people in the tech industry are well aware of the issues involved, and the deficiencies in Malcolm’s multi-technology mix version of the NBN. It’s a bit hard not to, when a crucial piece of infrastructure we rely on day to day to make a living doesn’t work reliably. Political point scoring is one thing, but some of us need a reliable and fast internet connect to do actual work, and every dropout is a testament to the failure of Malcolm Turnbull, as former Communications Minister in charge of the NBN, and more broadly as a failed politician. Seems people outside the tech industry generally don’t care enough to make a difference. And computer nerds generally aren’t very good at explaining to the masses why certain tech issues are important.

It’s all just screaming into the void. Same with all the other technology-related failures of the LNP government — the failed census, robodebt, the terrible metadata retention scheme, the even worse AA Bill, MyHealthRecord…the list goes on. This government cannot computer.

A lot of this seems to have been forgotten, as the media focuses elsewhere, and the news cycle inevitably moves on.

But if we don’t look back at the past, we will never learn from it, and risk repeating the same mistakes.

Malcolm Turnbull might be gone from politics, but the forces in place that led to his failures, and to his undermining of the NBN, are still there. The hostility towards technology, and ill-informed policy resulting from this, is still there within the Coalition, ready to undermine other industries, like renewable energy.

Of course, the state of the NBN is a result of many factors going back decades. As is always the case with politics, shitfuckery abounds, on both sides. The fact the Coalition even got elected in 2013 under the leadership of Tony Abbott was the result of the Labor party imploding. Labor is responsible for an otherwise unelectable Tony Abbott managing to actually win an election, and set the stage for Turnbull to undermine the NBN as Communication minister. But the Coalition shitfuckery in this area was generally worse than Labor’s when it comes to Australia’s broadband infrastructure — from John Howard’s fateful decision to privatise Telstra without structural separation, a series of incredibly incompetent Communications and Information Technology ministers (remember international laughing stock Richard Alston?), George Brandis introducing Metadata retention then spectacularly failing to explain metadata in a TV interview, and a series of draconian surveillance and anti-encryption laws mis-characterised as “national security”.

So while the state of the NBN might not be entirely Turnbull’s fault, he had a huge influence with his terrible decision to abandon Labor’s plan of universal optic fibre and change to a ridiculous “multi-technology mix” relying on archaic copper cables.

The resulting problems caused by this decision is essentially Malcolm Turnbull’s main legacy. We should remember this fuck up, and apportion blame for it to Turnbull, in the hope that future politicians in similar position of power would be more likely to make better decisions.

But what really annoys me about the whole thing is how predictable it was. Every single knowledgeable telecommunications expert at the time was of the opinion that Turnbull’s multi-technology mix NBN was various combinations of inferior, stupid and insane. Of course, there were numerous clueless partisans commentators willing to defend the coalitions’s flawed broadband policy. And these people’s clearly wrong opinions were taken seriously by a credible media wanting to “both sides” the issue.

I have never understood the contortions journalists make in the pursuit of some misguided ideal of “objectivity”. The problems with this are manifest in a world where Donald Trump is president. But the problems have been there for a long time, and have consequences. We’ve seen climate change sceptics with no expertise or qualifications put up again qualified climate scientists to debate climate change. We get the views anti-vaccination crackpots with no credible evidence presented along with the advice from qualified medical professionals and scientists.

The last decade or so of Australian (and international) politics has exposed the flaws of this kind of media reporting. There seems to be a desperate need by political journalists for narrative — and to believe that the people they are reporting on in positions of power have character, intelligence, gravitas. Surely the vagaries of our elected political leaders should have dispelled this myth by now? Our politicians are mostly power hungry grifters driven by a desire for fame, power, self aggrandisement and self enrichment, rather than any real desire for public service.

The political careers of both Malcolm Turnbull, and his predecessors as Prime Minister, Tony Abbott, benefited greatly from a credulous media. I’m not really talking about the right wing propaganda machine of New Corp, which is a whole other shitshow. People fed a diet of Murdoch bullshit are obviously going to be ill-informed.

It’s annoying that even more credible media outlets largely recycle press releases, or just uncritically report the utterances of politicians as news, even if it is plainly untrue or non-sensical. There was a hell of a lot of that going around at the time regarding the NBN.

Looking back to 2013, it’s clear now the LNP under Tony Abbott had no credible plan for a decent broadband network. “Better, faster, sooner” my ass. They wanted to do what they could to undermine the NBN — their NBN policy was just a weak exercise in plausible deniability.

Yet, it was reported in the media as if it was a credible communications policy, and after the election, Malcolm Turnbull was taken seriously as communications minister.

I think there was a sense at the time that after the Labor party imploded while in office, it kind of ruined the usual subtext of political reporting that these were the great men and women of history, and made it look more like high school bickering. So there was a desperate need amongst reporters to present the Coalition a sensible and responsible alternative. But Tony Abbott was opposition leader at the time, and he was neither. We certainly didn’t get sensible or responsible government with Abbott as Prime Minister. We got knights and dames, knighthoods for Prince Phillip, raw onion eating, and lots of other weirdness, but no sign of good government.

The entire country breathed a great sigh of relief when Abbott was deposed by Turnbull. The weirdness and delusion was gone, but more importantly, here, finally, was a Prime Minister that looked and acted the part. Turnbull looked like a Prime Minister. He spoke like a Prime Minister. His background make it seem that, on paper at least, he would make a good Prime Minister. And he enjoyed some initial popularity. That soon faded, however, as it became increasingly apparent he didn’t have much more idea of what he was doing than his predecessors.

It’s deeply weird to me that even when a politician conclusively demonstrates they have no idea what they’re talking about, not idea what they’re doing, and no ideas in general, they are still taken seriously by political journalists.

I think Turnbull was particularly interesting in this regard, because he looked and sounded the part so well. He was the ultimate cosplay as Prime Minister of Australia. And before that, he got to cosplay as Communications minister. Unfortunately, the media mistook his pathetic cosplay broadband policy as something that made sense, when every expert in the country was screaming that it did not.

Politicians are human beings after all. Being elected doesn’t automatically give them new skills or talents. Most at some point in their lives were second rate suburban accountants and lawyers. They’re all essentially cosplaying as politicians. (Except for Barnaby Joyce, who cosplays as a farmer. One of the more amusing owns of Barnaby Joyce I’ve seen on Twitter involve references to him being a suburban accountant before entering politics. But he wears Akubra hats and R.M Williams boots whenever there’s cameras likely to be around, trying to look rural. Seems to work though. Especially on gullible journalists.)

Obviously, looking like a Prime Minister does not guarantee success as one. Neither does being an investment banker or lawyer. On paper, Turnbull’s background makes it seem like he might actually make a good Prime Minister. In reality, it did not. Success in one field does not guarantee success in another. Donald Trump is another obvious example of that. Although it increasingly seems like Trump’s business success was largely a fiction of his own creation. I wonder how much of Turnbull’s business success is due to skill and talent, and how much to white male privilege and dumb luck? His supposed business acumen didn’t help when it came to building a world class broadband network. And it certainly didn’t help him handle the ideological and personal divisions within his own party that ultimately cost him his job as PM.

The NBN rollout is entering its last stages. Regardless of who wins the election, it’s unlikely it will get better any time soon. Turnbull’s terrible switch from fibre to a multi-technology mix have crippled it for years to come, without a great upgrade path. Eventually there will be no choice but to replace the remaining copper-based parts of the network with optic fibre, at great expense, but we’re stuck with what we’ve got for now.

So Malcolm Turnbull looked the part, sounded the part, and had a background that suggested he might be good as both Communications Minister, and then Prime Minister. But he wasn’t. He was terrible at both. The resulting NBN is a mess. And his political career ended with an incredibly messy coup by Peter Dutton (of all “people”!).

Turnbull’s career, and the NBN, provide an instructive look at what what promised, and what was eventually delivered. Neither was successful. Signs of problems with both were there from the start. (Problems that weren’t particularly prominently address by political journalists at the time.)

The LNP promised an NBN that would be fast, affordable, sooner. It’s none of that. Outside of the fibre parts of the network, it’s slow and prone to dropouts. As the NBN rollout finishes up, Australia continues to drop in global speed ranking — down from 50th to 62nd.

None of that has featured in this election campaign. As far as the NBN goes, the fact the coalition made a mess of it seems to have been forgotten.

It’s as if the NBN issue has been neutralised now that Turnbull is gone. But while Scott Morrison might not have had much direct involvement in screwing up the NBN (he was busy being cruel to asylum seekers) the same political forces are still there, and we shouldn’t forget Morrison is now leading the government responsible for screwing up the NBN. Since he’s only been PM for 8 months, he doesn’t have much of a record to run on, going into this election. But the political coalition he is notionally leading has been in government for 6 years, and in that time they have achieved little — apart from screwing up the NBN.

The government is asking to be re-elected but not providing much in policies. Morrison is running essentially on a platform that he is not Bill Shorten, nor is he Malcolm Turnbull. Or Tony Abbott.

Meanwhile, the climate is changing, and we’ve had six years of inaction, while the Coalition government worked to undermine renewable energy the same way they’ve undermined the NBN.

Remember, every time your internet connection craps out, that’s the legacy of Malcolm Turnbull.

With that ranting out of my system, I should now post this thing before my internet connection craps out. Again.

Wayward Windows

2018-12-18T00:00:00+11:00

In some occasional but disturbing fever dreams I consider buying another Windows device.

Then I power up one of my old Windows laptops and any such desire rapidly fades.

Waiting for an old Windows machine to boot up has a way of focusing the mind on the deficiencies of Windows as an operating system. And the ever-increasing boot times provide more and more time to contemplate the many and varied ways in which Windows disappoints and annoys.

One possible way to prevent oneself from impulse buying anything online is to try and do it on an old, underpowered Windows machine. By the time it boots, you’d have forgotten what you were going to buy. By the time a a browser loads (and then probably updates itself) you’ve most likely gone away to do something else, and forgotten why you powered the thing on in the first place.

I don’t use Windows much anymore — my development work these days is done mostly on Linux, sometimes a Mac.

So booting up one of my old Windows machines is inevitably an exercise in frustration. The frustration of the long boot up time, the endless hard drive thrashing before being presented with a useable desktop. Then there’s Windows Update. The slowdown to the point of being unusable, as Windows installs updates in the “background”. Windows Update increasingly seems like punishment for using Windows sparingly — the less you use the device in questions, the longer the wait for it to update, and the less useable the device is in the meantime. Yes, I get that security is important, but all this is hardly conductive to getting anything useful accomplished.

Of course, the two ageing Windows laptops I have are hardly representative of modern Windows machines — one a Windows 7 era machine, the other Windows 8, both with mechanical hard drives, and both now running Windows 10. But they were both powerfull machines for their time — both Core i7 processors (albeit many generations old), with plenty of RAM. It’s most likely the mechanical hard drives that are the main performance drag — it really is astonishing the difference solid state drives make to performance.

Though I do still wonder if the performance has to be this bad on older hardware? I wonder if any developer in any position to influence any operating system development still encounters machines with mechanical hard drives? Probably not, so either through unconscious bias or intentional design, OS development now assumes SSD hardware, and therefore modern operating systems run far more crappier than they should on older hardware.

Either way, the crappiness of Windows running on old hardware really doesn’t inspire confidence at best, and moreover acts as a deterrent to buying any new hardware with the intention of running Windows on it. Especially pricier, upper end hardware like Surface devices. Sure, they’ll probably be nice and performative initially. But how long before the usual Windows software rots sets in? How long before the shiny new device of desire becomes another laggy mess?

Nexus redux

2018-07-24T00:00:00+10:00

OK, so my phone actually got fixed. Finally. And somewhat surprisingly.

My Nexus 5x phone developed a defect a few months ago — the dreaded, and apparantly common, bootloop issue. It suddenly started rebooting, repeatedly and constantly, to the point it became unusable. LG Australia initially rejected my warranty claim. Their opening gambit was to try and fob me off by telling me my phone was an overseas model or grey import. It’s not. It is an Australian model purhased outright at a local JB Hi Fi store.

Finally, after repeated warranty claims, and uploading multiple copies of my receipt, they agreed to repair the phone. But I wasn’t expecting much when I posted the phone to their repair centre.

I’d come across a number of forums on whirlpool with horror stories of LG warranty issues, and some reports of LG refusing to repair bootlooping Nexus 5X devices. Excuses like falsely claiming water damage seemed common. My phone was otherwise in great condition, certainly no water damage — not even a scratch on the screen.

Perhaps this was why I did end up with a functional and repaired phone. They finally honored their warranty obligations.

Which is great, but because the whole thing dragged out for so long, I had to get a replacement phone anyway. And I’m not sure I trust the Nexus 5X anymore as my main phone.

The replacement phone I settled on was a Moto G5 plus.

It’s a mid-range phone, but does what I need it to.

I actually like it better than the Nexus. I think it’s an objectively better phone.

They’re both midrange phones with fairly stock Android.

But I think there’s more attention to detail with the Moto. It feels a little bit more refined. There are some small changes to the standard Android OS, and for the most part they are worthwhile changes. In particular, the notificaitions and home screen are really well thought out on the Moto.

I prefered the location of the fingerprint sensor on the Nexus — it perfectly located on the back. But this is a minor thing. The front sensor on the Moto is fine.

My main annoyances with the Nexus were the buttons. Clearly some corners were cut (hence the common bootloop issues). In particular, the power and volume buttons felt cheap and spongy. They were also overly sensitive, which was particular annoying in combination with the fact they were still functional even with the device “locked”. So often just picking up the phone meant an unwanted volume adjustment.

The vibration and synaptics were also felt weak and insubstantial.

All in all, the Moto feels like a more solid and well made phone, so I’m sticking with it. The repaired Nexus will be my backup/dev device. Assuming it doesn’t start bootlooping again.

LG refuses to honour warranty after my Nexus 5X develops bootloop issue

2017-12-23T00:00:00+11:00

This is the story of the unexpected death of my Nexus 5X — a phone taken too early by LG’s dodgy manufacturing and the inexorable march of planned obsolescence.

After my phone started randomly rebooting itself one night, I did some googling. Turns out this is not an unusual issue with the Nexus 5X model, which is made by LG. It is often referred to as the “bootloop” issue. What’s more, this is not the only model with this problem — it also affected some previous LG phones. There was even a class action lawsuit in the US.

While apparently LG has never officially stated what the root cause of this issue is, online speculation points towards either a flaw with the CPU design, or bad CPU soldering work.

Design flaw or manufacturing flaw, either way, seems like something that should be covered by some combination of either manufacturer’s warranty or Australian consumer law.

The phone was 15 months old and in good condition appearance-wise — no drops, dents, scratches or water damage. While 15 months isn’t exactly a short time to own a phone these days, there seems to be a general consensus that a phone should last at least two years. According the LG website, this phone comes with a 24 months warranty.

I still had the receipt (and original packaging) so I took it back to the JB Hi-Fi store I’d bought it from, but they weren’t much help.

So I lodged a warranty repair request through the LG Australia website. After a few days I received an email saying my device was an overseas model so LG Australia didn’t want to have anything to do with it.

This made me so angry.

I had purchased the phone in-store at JB Hi-Fi. My understanding was that JB only sold Australian stock. I went to a store and asked a salesperson, who confirmed that JB only sold Australian stock. So some corporation here are being cunts about this. Either JB are selling overseas or grey import stock and claiming it’s Australian stock, or LG are full of shit.

I suspect the latter is more likely.

I’m assuming that this is essentially an opening gambit on the part of LG, who are trying to weasel their way out of their warranty obligations, as well as their obligations under Australian consumer law.

Well, even though this meant I had to buy a new phone, I’m not ready to give up just yet. There is now a principle at stake. LG sold me a dodgy phone, so they should at least fix it.

To be continued…

Death by bootloop of my Nexus 5X

2017-12-21T00:00:00+11:00

“I was talking to my sister today, and guess what she said to me…”

Then silence, as my girlfriend paused for dramatic effect. At least that’s what I thought it was.

I was on the phone to my girlfriend. At least I thought I was.

I looked at my phone. The screen was off, and the phone didn’t respond to touches or key presses. Hmmm…couldn’t be the battery, I had just taken it off the charger. I held down the power key. It started to reboot.

I called her back.

My girlfriend once again started to regale me with the latest installment of interpersonal drama between herself and one of her sisters. Then the phone went dead again. I looked at the screen. It was rebooting again. Hmmm…had I inadvertently held down the power key, while steeling myself to hear about the latest installment of interpersonal drama between my girlfriend and one of her sisters? Possibly, but I didn’t think so.

Had my Nexus 5X become self-aware enough to know it no longer wanted to hear about my girlfriend’s interpersonal drama with her sisters? Possibly, who knows what Google’s machine learning algorithms are capable of, but I didn’t think so.

A third attempt at a phone conversation was also rudely interrupted by a spontaneous reboot, at which point it was clear something was going on with my phone. Had the force of my girlfriend’s sibling-induced rage remotely fried my phone? Possibly, but I didn’t think so.

We gave up on having a phone conversation that night, and I never got to find out what her sister had said to get her so riled up that day — by the time we next spoke she had moved on to another drama with another sister.

I didn’t try to make any more calls that night, but soon noticed that even when I put my phone down and didn’t use it, after a while it would still spontaneously reboot. And I began to get the impression the time between reboots was getting shorter and shorter.

At that point it was clear there was something wrong with my phone. I then realised that I didn’t have the photos on the phone backed up. I’d had the phone for around 15 months. Any other data of any importance would be synced to a cloud account, but I had deliberately turned off all cloud sync on my photos. So I had over a year’s worth of photos on a phone that was malfunctioning. Nothing earthshatteringly important, but still, stuff I’d like to retain.

I found a USB-C cable, then spent the rest of the night copying photos from the phone to my laptop — in batches, in between reboots. (I have never been more grateful for USB 3 transfer speeds!)

It was a good thing I did too. By the next day, the phone wouldn’t even boot up at all. It would reboot itself constantly, before it had even finished booting up. Pretty soon it was rebooting at the splash screen.

Great, so my phone was fucked. After just over a year. Planned obsolescence strikes again. It’s also the second Nexus branded device I have owned that failed after a relatively short time.

VoidyBootstrap version 2.1 released

2017-01-31T00:00:00+11:00

VoidyBootstrap is a Pelican theme based on Bootstrap 3 that I hacked together when I first started using Pelican, and have been all too sporadically updating since.

Version 2.1 has now been released.

This release is mostly just updates to dependencies (Bootstrap, Font Awesome, etc) and some various minor fixes and tweaks.

The most significant changes are for compatibility with Pelican 3.7 (and 3.6).

I don’t believe any of the changes in this release break backward compatibility, except for a few things related to the default sidebar implementation.

In VoidyBootstrap, the sidebar is optional, and the default is single column mode. The theme allows for an optional sidebar to be configured by using the SIDEBAR option to specify the filename of a Jinja template that will constitute the sidebar contents. A basic sidebar template is included as a starting point, but a custom template can be created and used instead.

For those using the built-in default sidebar, there’s a few things to be aware of if upgrading from version 2.0 to 2.1.

Firstly, there is no longer a tag cloud by default. This is because tag cloud support has been moved from Pelican itself to a plugin in Pelican 3.7. It therefore seemed the most logical option for the theme was to remove the tag cloud, and provide an option to enable it for those who want to configure the tag_cloud plugin. You can of course roll your own tag cloud template and incorporate that into the sidebar (or elsewhere). I’ve provided a tag cloud template in sb_tagcloud.html, which can be incorporated into the default sidebar using the following setting in pelicanconf.py:

CUSTOM_SIDEBAR_MIDDLES = ("sb_links.html", "sb_taglist.html", )

The sb_links.html template fragment is only required if the LINKS setting is being used. Which is the other breaking change in this release: the LINKS setting is now also disabled by default, and can be enabled by incorporating sb_links.html as shown above.

Thanks to everyone who contributed pull requests.

Upgrading to Pelican 3.7

2017-01-16T00:00:00+11:00

The new year brings with it a new release of Pelican. Well, actually, Pelican 3.7 was released last month, but it’s in these first few weeks of the new year that I find myself with some spare moments to devote to trying out the latest version — a task somewhat overdue, having paid little attention to Pelican since version 3.5. So in this post I’m going to have a look at changes in Pelican since version 3.5 that might be relevant when upgrading an existing website.

Pelican changes

In general Pelican itself tends to be fairly backwards compatible, so often all that’s required is to upgrade the Pelican version in whatever virtual Python environment you might be using. However, there are often interactions between themes and plugins that might break things when upgrading.

There were a few things that affected me and the combination of theme and plugins that I was using after upgrading to Pelican 3.7.

One is that the uppercase PAGES context variable available to themes has been removed in favour of the lowercase pages. Unfortuantely, many themes use the uppercase version and therefore need to be updated before they will work properly with Pelican 3.7.

Another is that tag cloud support has been removed from Pelican itself and moved to the tag_cloud plugin. So if your site has a tag cloud, you’ll now need to configure the plugin.

On a side note, I just recently noticed the sitemap plugin now supports excluding URLs from the sitemap via regular expressions, which I find incredibly useful, since I don’t want tag or category pages in my sitemaps.

Pelican site configuration

While there’s a variety of possible workflows with Pelican (it being a command line tool after all) I assume it’s fairly typical to have static sites originally created using the Pelican quickstart script. This means there’s a Makefile (or Fabfile) that was created with a previous version of Pelican that won’t have enhancements or features present in recent versions.

So as I usually do when undertaking a Pelican upgrade (because I’m kind of anal retentive about such things), I ran quickstart to create a dummy skeleton site, then compared it with previous versions, to see if there is anything significant enough to incorporate into my site’s existing configuration.

Only the Makefile had any differences of note.

A diff between a version 3.5 and 3.6 Makefile gives (abridged, differences in echo output ommitted):

34a35,39
> RELATIVE ?= 0
> ifeq ($(RELATIVE), 1)
>     PELICANOPTS += --relative-urls
> endif

...

72a80,87
> serve-global:
> ifdef SERVER
>     cd $(OUTPUTDIR) && $(PY) -m pelican.server 80 $(SERVER)
> else
>     cd $(OUTPUTDIR) && $(PY) -m pelican.server 80 0.0.0.0
> endif
>
>
81,82c96
<     kill -9 `cat pelican.pid`
<     kill -9 `cat srv.pid`
---
>     $(BASEDIR)/develop_server.sh stop
110c124
< .PHONY: html help clean regenerate serve devserver publish ssh_upload rsync_upload dropbox_upload ftp_upload s3_upload cf_upload github
---
> .PHONY: html help clean regenerate serve serve-global devserver publish ssh_upload rsync_upload dropbox_upload ftp_upload s3_upload cf_upload github

So the main changes are support of the new relative URLs option, along with some changes to running the development server.

A diff of the version 3.6 and 3.7 Makefiles gives:

115c115
<     s3cmd sync $(OUTPUTDIR)/ s3://$(S3_BUCKET) --acl-public --delete-removed --guess-mime-type
---
>     s3cmd sync $(OUTPUTDIR)/ s3://$(S3_BUCKET) --acl-public --delete-removed --guess-mime-type --no-mime-magic --no-preserve
124c124
< .PHONY: html help clean regenerate serve serve-global devserver publish ssh_upload rsync_upload dropbox_upload ftp_upload s3_upload cf_upload github
---
> .PHONY: html help clean regenerate serve serve-global devserver stopserver publish ssh_upload rsync_upload dropbox_upload ftp_upload s3_upload cf_upload github

There was also a tweak to develop_server.sh in version 3.7:

69c69
<   cd $OUTPUTDIR
---
>   mkdir -p $OUTPUTDIR && cd $OUTPUTDIR

Android Tabs: How to use TabLayout

2015-08-16T00:00:00+10:00

This post presents a simple example of using TabLayout from the Android Design Support Library.

It’s the second part in a series about implementing tabs on Android without using any APIs deprecated in API level 21. In the previous post I looked at PagerTabStrip, which works well enough. However, TabLayout provides a look and feel more in keeping with Material Design conventions. In this post we’ll convert the example from the previous post to use TabLayout instead of PagerTabStrip.

The first thing we need to do is include the Design Support Library in the dependencies of the app’s build.gradle file:

compile 'com.android.support:design:22.2.0'

In the activity layout (activity_tab_layout.xml) we place a TabLayout widget and a ViewPager:

<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    android:orientation="vertical" android:padding="4dip"
    android:gravity="center_horizontal"
    android:layout_width="match_parent" android:layout_height="match_parent">

    <android.support.design.widget.TabLayout
        android:id="@+id/tabs"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        app:tabMode="fixed"
        app:tabGravity="fill" />

    <android.support.v4.view.ViewPager xmlns:android="http://schemas.android.com/apk/res/android"
        xmlns:tools="http://schemas.android.com/tools"
        android:id="@+id/pager"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        tools:context="net.voidynullness.android.tabitytabs.TabLayoutActivity">

    </android.support.v4.view.ViewPager>

</LinearLayout>

There’s a number of useful attributes for TabLayout.

Above, we set tabMode to fixed to show all tabs. The other alternative is scrollable, where the tabs can be scrolled (similar to PagerTabStrip).

tabGravity determines alignment of tabs. It can be set to fill to space out tabs to fill available space, or centre to cluster the tabs in the middle of the TabLayout.

If we wanted to we could programmatically add tabs using the addTab() and setText() methods:

tabLayout = (TabLayout) findViewById(R.id.tabs);
tabLayout.addTab(tabLayout.newTab().setText("Tab 1"));
tabLayout.addTab(tabLayout.newTab().setText("Tab 2"));
tabLayout.addTab(tabLayout.newTab().setText("Tab 3"));

But we’re not going to do things that way for this example.

Instead, we will create a hosting activity (that subclasses AppCompatActivity) which will use the above layout, and wire up the ViewPager to the TabLayout. The ViewPager will take care of fragment switching for us (TabLayoutActivity.java):

package net.voidynullness.android.tabitytabs;

import android.os.Bundle;
import android.support.design.widget.TabLayout;
import android.support.v4.view.ViewPager;
import android.support.v7.app.AppCompatActivity;


public class TabLayoutActivity extends AppCompatActivity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_tab_layout);

        TabLayout tabs = (TabLayout) findViewById(R.id.tabs);
        ViewPager pager = (ViewPager) findViewById(R.id.pager);
        TabsPagerAdapter adapter = new TabsPagerAdapter(getSupportFragmentManager());

        pager.setAdapter(adapter);
        tabs.setupWithViewPager(pager);
    }
}

No changes are required to our FragmentPagerAdapter (TabsPagerAdapter) or sample fragment class (PageFragment) from the previous post, so we can reuse those without changes. (See the previous post for details and implementation of these two classes).

The above activity’s onCreate() instantiates a new TabsPagerAdapter, which is our subclass of FragmentPagerAdapter, and passes it to the ViewPager, which handles most of the work of switching between tabs and fragments. To configure the TabLayout we just need to call setupWithViewPager() and pass in the ViewPager we just configured. setupWithViewPager() is a convenience method that sets things up for the common use case of letting a PagerAdapter configure the tabs. (The PagerAdapter in this case being TabsPagerAdapter.)

The result is a tabbed layout with a reasonably Material Design compliant look:

The complete source code is on BitBucket.

Android Tabs: Using PagerTabStrip with ViewPager

2015-08-12T00:00:00+10:00

The release of Android API level 21 (Lollipop) deprecated a number of APIs — in particular, anything to do with ActionBar tabs.

This means a common way of implementing tabbed navigation is now discouraged, but things have not been particularly clear as to what replaces these deprecated APIs. Of course, deprecated doesn’t mean unusable. Deprecated though they are, ActionBar tabs will still work.

But anyone starting a new long term Android project might reasonably baulk at firing up the Android Studio “New Project” code wizard only to be greeted with a whole lot of deprecation warnings in the generated code for anything with tabs. Even if the resulting code does actually work. For now.

So what are our options if we want tabs, but don’t want to use any of these deprecated APIs?

Roll your own tabs implementation (Obviously not helpful to any developers dealing with onerous deadlines.)
Use a third party implementation.
Incorporate or adapt the implementation of tabs in either the Android SDK Sliding Tabs sample or the Google iosched app.
Use the venerable PagerTabStrip.
Use TabLayout from Google’s newish Android Design Support Library.

I’ll explore the last two options in this tutorial. In this post I’ll show a simple implementation of tabs using PagerTabStrip. In the next post I’ll look at TabLayout.

PagerTabStrip works in conjunction with ViewPager, and plays well with AppCompatActivity (although the results aren’t exactly Material Design friendly).

For this example project we’ll use AppCompatActivity as our Activity class, and Theme.AppCompat.Light.DarkActionBar as the application theme.

We’ll start with the layout for our activity (activity_pager_tab_strip.xml), which in this case will just be a ViewPager containing a PagerTabStrip:

<android.support.v4.view.ViewPager xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools"
    android:id="@+id/pager"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context="net.voidynullness.android.tabitytabs.PagerTabStripActivity">

    <android.support.v4.view.PagerTabStrip
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="top"/>

</android.support.v4.view.ViewPager>

The layout_gravity is set to “top” so that the tabs are displayed at the top of the ViewPager.

The activity code (PagerTabStripActivity.java) is fairly straightforward:

package net.voidynullness.android.tabitytabs;

import android.os.Bundle;
import android.support.v4.view.ViewPager;
import android.support.v7.app.AppCompatActivity;

public class PagerTabStripActivity extends AppCompatActivity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_pager_tab_strip);

        TabsPagerAdapter adapter = new TabsPagerAdapter(getSupportFragmentManager());
        ViewPager pager = (ViewPager) findViewById(R.id.pager);
        pager.setAdapter(adapter);
    }
}

There’s not much going on in the Activity — it’s basically just configuring the ViewPager by giving it a suitable adapter object. In this case the adapter class is TabsPagerAdapter, which is our subclass of FragmentPagerAdapter (see below). Notice that we need to pass a FragmentManager to the adapter, and here we’re passing the support library fragment manager, since we’re using AppCompatActivity.

TabsPagerAdapter is a subclass of FragmentPagerAdapter, and it’s here that the fragments corresponding to each tab page are instantiated, and tab title text is defined (TabsPagerAdapter.java):

package net.voidynullness.android.tabitytabs;

import android.support.v4.app.Fragment;
import android.support.v4.app.FragmentManager;
import android.support.v4.app.FragmentPagerAdapter;


public class TabsPagerAdapter extends FragmentPagerAdapter {

    public TabsPagerAdapter(FragmentManager fm) {
        super(fm);
    }

    @Override
    public Fragment getItem(int position) {
        return PageFragment.newInstance(position + 1);
    }

    @Override
    public int getCount() {
        return 3;
    }

    @Override
    public CharSequence getPageTitle(int position) {
        return "TAB " + (position + 1);
    }
}

It’s about as minimal as a FragmentPagerAdapter can be. At the very least, subclasses of FragmentPagerAdapter must implement getItem() and getCount().

getItem() is expected to return the fragment for the specified position. Internally, FragmentPagerAdapter calls getItem() to instantiate fragments, and uses the FragmentManager passed into the constructor to manage these fragments.

getCount() should return the total number of fragments.

We’re also overriding getPageTitle(), which is used to set the text for each tab position.

So we have our ViewPager and FragmentPagerAdapter, now we need a fragment to actually display something. We’ll call our simple placeholder fragment class PageFragment. It will just display a text string on the screen to indicate which fragment “page” is currently selected (PageFragment.java):

package net.voidynullness.android.tabitytabs;

import android.support.v4.app.Fragment;
import android.os.Bundle;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.TextView;


public class PageFragment extends Fragment {
    private static final String ARG_PAGE_NUMBER = "page_number";

    public PageFragment() {
    }

    public static PageFragment newInstance(int page) {
        PageFragment fragment = new PageFragment();
        Bundle args = new Bundle();
        args.putInt(ARG_PAGE_NUMBER, page);
        fragment.setArguments(args);
        return fragment;
    }

    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
                             Bundle savedInstanceState) {
        View rootView = inflater.inflate(R.layout.fragment_page_layout, container, false);

        TextView txt = (TextView) rootView.findViewById(R.id.page_number_label);
        int page = getArguments().getInt(ARG_PAGE_NUMBER, -1);
        txt.setText(String.format("Page %d", page));

        return rootView;
    }
}

The newInstance() idiom of a static factory method is used to create and initialise new fragments.

As shown above, our fragment adapter (TabsPagerAdapter) creates PageFragment fragments by calling the static newInstance() method and passing in the page/tab number of that particular instance. Inside newInstance(), the page number is saved as an argument bundle.

In onCreateView(), the TextView contents are updated to display the page number, which is retrieved from the arguments.

The corresponding layout file for our PageFragment class simply contains a single TextView to display the page number (fragment_page_layout.xml):

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent"
    android:layout_height="match_parent" android:paddingLeft="@dimen/activity_horizontal_margin"
    android:paddingRight="@dimen/activity_horizontal_margin"
    android:paddingTop="@dimen/activity_vertical_margin"
    android:paddingBottom="@dimen/activity_vertical_margin"
    tools:context="net.voidynullness.android.tabitytabs.PageFragment">

    <TextView android:id="@+id/page_number_label"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:textAppearance="@android:style/TextAppearance.Large"
        android:gravity="center"
        android:layout_centerVertical="true" />

</RelativeLayout>

The result is three fragments, that can be switched by swiping or selecting the appropriate tab:

The complete source code for this example is on BitBucket.

In the next post, I’ll take a look at an alternative to PagerTabStrip — TabLayout from the Android Design Support Library — which provides similar functionality but is more in keeping with Material Design conventions.

Big Government Puppies

2015-08-02T00:00:00+10:00

When training puppies, apparently consistency is really important.

If you want your puppy to stay off the furniture, then they should never be allowed on the furniture. Ever. Mixed messages don’t work. Puppies cannot understand nuance. (Come to think of it, neither can politicians, it seems.) If the puppy ever jumps on the furniture, they should be gently but firmly removed. Every time. So that they eventually realise they are not allowed on the furniture.

If, in moments of weakness, they puppy is ever allowed on the furniture, it undermines the message — and the owner’s authority. The dog will eventually realise its owner is weak, and can be exploited. They are then more likely to keep pushing their luck, constantly jumping on the couch in the hope they may be allowed to remain there. They might even get the idea the furniture is their territory, and get aggressive if anyone tries to move them.

Being more of a cat person, I have no idea if this is true.

But in any case, it strikes me that this is a weirdly apt analogy for the relationship between us, and the politicians who supposedly represent our interests.

Politicians, much like mischievous puppies, will always try to push their luck. They will try to get away with as much as they can. Much like we shouldn’t allow our pets to shit on the carpet and have free reign over our furniture, we shouldn’t allow our politicians to blithely expand their powers, remove prudent oversight, or eschew accountability.

For an example of what happens when we do, we just need to take a look at what’s been happening in the Australian telecommunications industry, especially since the Abbott government was elected.

First, this government destroyed all that was good about the NBN. While we can speculate about why exactly they might have wanted to do this (was it for the benefit of Rupert Murdoch’s media empire?), I suspect they did it partly out of ignorance — not understanding the social and economic benefits a universal fibre network would bring — but mostly just because they could. They burned the NBN and scorched the earth afterwards, just to see if they could get away with it. And they did get away with it. Because we didn’t push back strongly enough.

Of course, anyone thinking that was the end of the Coalition’s interest in screwing up the tech industry would have been profoundly naive. Especially with an Attorney-General like George Brandis, whose ego and sense of infallibility is matched only by his complete ignorance of the telecommunications industry and anything technical.

It was only the beginning.

After a hobbled NBN, we got the controversial mandatory data retention. There is no evidence that data retention is effective in reducing crime or terrorism, it possibly makes us less safe because our metadata will be an irresistible target for hackers and identity thieves, and it will be really expensive. Not to mention the burden and red tape it imposes on the telcos, and the inevitable scope creep

But this government didn’t stop there. After that was the website blocking laws, presumably to appease large corporate donors to the Liberal party, like Village Roadshow. Once again, despite no evidence website blocking will be effective at its supposed purpose of preventing illegal downloading — and considerable international evidence that it won’t. And once again adding costs and red tape for the telcos.

They came for our NBN, and we did nothing that stopped them, so we missed out on fast fibre broadband. Then they came for our privacy, and we did nothing that stopped them, so we got data retention. Then they came for our websites, and we did nothing that stopped them, so we got website blocking and internet censorship.

Now, Brandis is once again meddling in areas outside both his portfolio and expertise — with predictably dire results.

The Attorney-General’s Department is planning to introduce more stupid national security legislation targeting telecommunications companies (The Telecommunications and Other Legislation Amendment Bill 2015) and the telcos are really unhappy. If these laws pass, we will have a guy who doesn’t know what metadata means, and who is confused about even the most basic technical details of the internet — like the difference between DNS and IP addresses — being able to dictate to ISPs what kind of technology they are allowed to use, and to interfere at will with the day to day operations of telcos by requesting arbitrary documents and information.

The proposed legislation exhibits all the typical problems of government overreach. These laws are intrusive and will be expensive to implement. The legislation is vague and there is no justification for these new government powers — it’s unclear what problems these laws are trying to solve. There has also been too little consultation with the industry

Even at the best of times, politicians and government seem to act on the mistaken belief that there is no problem that can’t be solved with legislation — and without unintentional consequences. IT security is a difficult problem, but it’s not one that gets any easier with the Attorney-General’s department breathing down the neck of the entire industry. IT security certainly doesn’t get any easier with technically clueless government bureaucrats trying to dictate technology choices. And even when governments act with the best of intentions — which is not the case here — there is always the risk they will make something worse. Which is why we must be vigilant (and suspicious) of all attempts at governments expanding their powers, and curtailing ours.

If politicians are not put in their place by the electorate, they end up making stupid laws like this. We end up with a situation where politicians end up thinking they are better placed to dictate security than computer security experts.

The telcos are now pushing back, but I suspect their protests are a case of too little too late.

Like a puppy who was constantly allowed to sleep on the couch, and so grows up to be a dog expecting that privilege, they government now knows it can get away with messing with the telecommunications industry in whatever way it sees fit — and it’s not going to give up that privilege without a fight. It’s certainly not just going to back down at the first sign of resistance.

A dog owner who never properly trained his puppy to stay off the couch — who perhaps only meekly objected when the puppy jumped on the couch — has only themselves to blame if the puppy grew up into a dog that refuses to get off the furniture.

So we — as the consumers who will end up paying for all of this — along with the telcos, have only have ourselves to blame. Because we didn’t fight back strongly enough.

With the possibly exception of iiNet, the telcos put up weak resistance to the data retention and website blocking laws, which gave the government more confidence in introducing further intrusion into their operations.

It seems when it comes to combating government overreach, we can’t afford to sit back, and pick and chose our battles. We need to fight constantly and consistently. If we want our puppies to stay off the couch, we need to be consistently firm with keeping them off the couch. So with our meddlesome politicians, we need to constantly fight them off from taking away our freedoms, civil liberties and digital rights. We can’t just think, “I have nothing to hide, so I don’t care about data retention”. Because as we have just seen, things like that tend to be the thin edge of the wedge. There is always much worse to come.

Attorney-General George Brandis is drunk on power, bloated of ego, and oblivious to his own ignorance. And he’s coming for the tech industry.

It’s time for all of us to stand up, menacingly brandish a rolled up newspaper in front of his face, and firmly and loudly say: “DOWN!”

“Bad dog, George. Bad. Get down! And stay down”

And if he doesn’t obey our wishes, he deserves a whack on the nose.

Mandatory protocol analysis to the node

2015-07-14T00:00:00+10:00

Gather round all ye geeks, wonks, activists, civil libertarians, conspiracy theorists — and let us all read between the lines of a job ad that was posted on Seek recently.

Let’s read between the lines of this job ad as we play everyone’s least favourite game of “guess which particular civil liberties and digital rights are under attack from the Australian government this month”?

The ad is titled “Software Developer - Protocol Analysis” and was posted (via a recruitment agency) on 12 June 2015. It’s a reasonably high paid role, located in “Central Melbourne”, for a “Federal Government Department”.

Turns out the Federal Government Department in question is the Attorney-General’s Department.

That would be Attorney-General George Brandis. Remember him? He’s the guy who really, really wants our metadata, without a warrant, and without knowing what metadata actually means.

His department is responsible not just for the undemocratic and misguided data retention laws, but also the laws threatening to put investigative journalists in jail for 10 years for doing their job.

Then there’s the government’s recently introduced website blocking laws, probably designed to appease their corporate donors.

With all that in mind, let’s take a look at some of the specific skills the Attorney General’s department is looking for.

Protocol analysis of IP network applications and services. This includes aspects such as reverse engineering of network protocols and traffic analysis

For those less technically inclined, a protocol is an agreed set of rules that networked devices use to communicate with each other. Think of it as a particular language computers use to talk to each other.

For example, BitTorrent is a protocol for peer-to-peer file sharing. Could it be this protocol that the Attorney-General’s department is most interested in analysing?

Demonstrated understanding of IP networks and protocols

They’re looking for a software developer with specific knowledge of networking protocols. And also experience with spying on network traffic:

Experience with network protocol analysis tools such as Wireshark, TCPDump etc

Both Wireshark and tcpdump are what’s known as packet analysers. They’re used to capture, store, and analyse network traffic. They basically a way of “spying” on network communication — not necessarily in a sinister way, but of course they can be used for devious purposes.

Packet analysers are essentially a diagnostic tool. I’ve used tcpdump occasionally in the past, when debugging low level networking code. Still, it’s a bit unusual to see it explicitly mentioned in a software development job ad. It’s more likely something an networking/security engineer might use.

Putting it all together, it paints a kind of sinister picture. Especially given the way the Abbott Government has been waging war on the internet. First data retention, then website blocking, now this. What the hell is this ridiculous government up to now?

Why does the Attorney-General’s department want a software developer with network spying skills? Come to think of it, why does the Attorney-General’s department even need a software developer at all?

To paraphrase Tom Waits: What’s George Brandis building in there?

OK, so maybe it’s nothing. Maybe it is all completely innocuous. Maybe there’s a legitimate reason George Brandis’s department needs a software developer.

Maybe the Attorney-General’s department just want a technical person on staff to repeatedly explain metadata to Brandis?

Or maybe George Brandis is now overcompensating for that embarrassing Sky news interview where he badly fumbled his explanation of metadata. So maybe now he’s constantly dropping the term “metadata” into conversation, in an effort to prove he’s finally learned what it means. But apart from coming across as awkward and try-hard, he’s still getting the context wrong, and thereby demonstrating that he still has no idea what it really means. Maybe Brandis is doing things like coming back from lunch and loudly proclaiming he enjoyed his hot-dog with added metadata. And maybe his staff are tired of explaining: “That’s not metadata, George, it’s mustard.” Maybe there is no one on his staff who can adequately explain to Brandis that mustard is a condiment, while metadata retention is an invasion of privacy. That mustard is delicious, while data retention tastes of government overreach, with a bitter aftertaste of privacy violation — and possible side effects of widespread identity theft.

Maybe it’s all mundane and innocent and there’s no agenda. But probably not.

It’s most likely this job has something to do with either mandatory data retention, the new website blocking laws, or possibly some new scheme attempting to combat illegal downloading.

Or maybe all of the above.

I can’t imagine any scenario where a role like this in the Attorney-General’s department won’t result in some combination of curtailing civil liberties and government overreach. But then maybe I’m just not as imaginative as the Abbott government.

How to add a list of recent posts to a Pelican blog

2015-06-25T00:00:00+10:00

List of Recent Posts

If you’re a blogger, and if your blogging platform of choice is Pelican (And if not why not? Pelican really is quite nice!), and if you’d like to have a list of recent posts somewhere in the page layout, but the Pelican theme you’re using doesn’t have built in support for this, it’s not too difficult to add, as long as you’re comfortable messing about with the Jinja templates of the theme in question.

The Jinja template code to accomplish this is reasonably straightforward, although you need to take into account that the articles variable means something different depending on which index page is being rendered. In index.html, articles is a list of all articles. In all other index pages (i.e. author, category, tag), it’s only a subset relevant to that index — the list of all articles is instead available as a variable called all_articles.

Here is an example that will render the 6 most recent posts as an unordered list.

<h4>RECENT POSTS</h4>
{% set recent = articles if not all_articles else all_articles %}
<ul id="recent-posts">
{% for a in recent %}
  {% if loop.index <= 6 %}
    <li>
      <a href="{{ SITEURL }}/{{ a.url }}">{{ a.title }}</a>
    </li>
  {% endif %}
{% endfor %}
</ul>

Place the above snippet in an appropriate place in your theme’s Jinja templates and style to taste.

Now With Added JSON

But what if — maybe, for some reason — you’d like to get details of recent posts in JSON format instead? Why would anyone want to do that, you ask?

Well, to provide a contrived example of using Pelican’s Template Pages functionality, for one thing.

I recently discovered the TEMPLATE_PAGES setting, and have been exploring its possibilities. It’s quite nifty. Yes, I used the word “nifty”. Yes, I probably need to get out more.

TEMPLATE_PAGES is also somewhat vaguely documented. It’s purpose is to basically enable creation of any kind of other custom content, besides articles and pages. The idea is you can add a template, which can have any kind of arbitrary content (and need not inherit from the theme’s base.html), and specify where the output should go.

Since the template can be pretty much anything, it doesn’t have to be HTML. We can generate JSON output, if we so desire.

Let’s say we were to create a file called recent.json in our theme’s templates directory, with the following contents:

{
  "recent": [
      {% set recent = articles if not all_articles else all_articles %}
      {% for a in recent %}
        {% if loop.index <= 6 %}
          { "url": "{{ SITEURL }}/{{ a.url }}",
            "title": "{{ a.title }}"
          } {% if not loop.last and loop.index < 6 %},
          {% endif %}
        {% endif %}
      {% endfor %}
  ]
}

Then we add this to our pelicanconf.py:

TEMPLATE_PAGES = {'recent.json': 'api/recent.json', }

TEMPLATE_PAGES is a dictionary mapping template to output. So in the above example, we’re telling Pelican to render the templates/recent.json template and place the output into output/api/recent.json (assuming the Pelican output directory is configured to be “output”). That file is now accessible on the /api/recent.json URL path in the usual way, and available for all sorts of AJAX mischief.

As an example, here’s some jQuery that consumes the above JSON output to render a list of recent posts dynamically:

$(document).ready(function() {
    $( "#payload" ).html("<p>...loading...</p>");
    $.getJSON("/api/recent.json",
            function(data) {
                var items = [];
                $.each( data.recent, function( index, val ) {
                    items.push( '<li><a href="' + val.url +'">' + val.title + '</a></li>' );
                });
                var newhtml = '<ul id="recent-posts">';
                newhtml += items.join( "" );
                newhtml += '</ul>';
                $( "#payload" ).html(newhtml);
            }
    );
});

Not that I’m in any way suggesting this method of generating a list of recent posts is preferable to the previous method (static generation via Jinja template). It is merely an example of what’s possible. Sensible use cases are left as an exercise for the reader…and possibly future blog posts.

Fail Faster

Another occasionally useful but vaguely documented Pelican setting is WRITE_SELECTED.

If you have a blog with a considerable amount of content, you may notice it takes entire seconds to generate your static site. Probably many, many seconds. Especially if Typogrify is turned on, which tends to slow things down a bit. If there’s really lots of content and/or you’re running on a really slow (virtual) machine, it might even take a minute or more to regenerate your site.

And the more content you write, the slower things will only get.

Seconds. Maybe a minute. In this world of instant gratification, who’s got time to wait seconds for a build?

Those seconds really add up when doing theme tweaks, or working on getting an article’s markup just right.

If you’re anything like me, you want The Machine to pass judgement on your substandard markup and declare it to be unparsable garbage as soon as possible.

Thankfully, Pelican has got you covered. If you’re just tweaking a page or two, you can temporarily use the WRITE_SELECTED setting in your pelicanconf.py to only output the page or pages you’re currently editing:

WRITE_SELECTED = ['output/blog/2015/01/01/path-to-blog-post/index.html',
                  'output/blog/pages/stuff/index.html',
                 ]

The slightly tricky thing is that WRITE_SELECTED apparently needs to be set to a list of full paths the the output files, including output directory. These will obviously depend on your URLs settings and output directory configuration.

With WRITE_SELECTED set only those files listed will be regenerated, which should make things substantially faster.

Just remember to remove the WRITE_SELECTED setting before publishing and uploading the final finished results, to make sure everything gets refreshed properly.

What’s New In VoidyBootstrap Version 2

2015-06-01T00:00:00+10:00

VoidyBootstrap is a theme for the Pelican static site generator.

This post is a summary of the main changes introduced in Version 2 of VoidyBootstrap.

As far as blog themes go, VoidyBootstrap is deliberately minimal in terms of design. It’s based on Bootstrap, and by default it just provides a standard Bootstrap look and feel. Nothing much has changed in this regard.

However, its main reason for existing is flexibility and customisation, and it’s here where I’ve made significant changes, some of which break backwards compatibility with previous versions. This is something I wanted to avoid, ideally, but in the end I decided it was justified. With these changes the result is a Pelican theme that is now even more flexible and customisable, with a more consistent naming convention for its many configuration options.

I never actually intended to undertake such an extensive update. When I started working on what would eventually become version 2, I just planned to update dependencies (Bootstrap, FontAwesome, etc) to the latest versions, and make a few other tweaks. But as often happens with these things, what started as a few small tweaks quickly escalated to into weeks of large scale hacking and refactoring. I ended up adding and changing so many things I felt I should call this release “version 2”. (For what it’s worth though, I don’t intend on making any more large scale changes like this. The version 2 series should be just small incremental updates — hopefully — but we’ll see…)

Overview

VoidyBootstrap started off as a custom theme for this blog, then gradually evolved into a stand-alone open source theme. Unsurprisingly, therefore, it reflects my requirements and is geared towards my Pelican workflow. VoidyBootstrap is admittedly a somewhat idiosyncratic Pelican theme, but it works for me, although of course your mileage may vary.

VoidyBootstrap came about out of my desire for a simple Bootstrap-based Pelican theme that was easy to customise. There are 3 main overriding goals I had with VoidyBootstrap:

It had to be flexible and customisable.
Despite its focus on customisation, it should be easy to set up. It should “Just Work” out of the box, with sensible defaults, and good documentation.
Customisation workflows should play nice with version control, making it possible to isolate changes and easily merge in upstream changes in the base theme.

Much of the changes in version 2 have been motivated by that last point, even though it may not be immediately obvious that’s the case.

Sidebar Changes

The most visible change in VoidyBootstrap version 2 is the sidebar — in that by default, there now isn’t one!

The sidebar is now optional, so VoidyBootstrap can now be used in either single column or 2 column mode.

For implementation reasons, the default is now single column without a sidebar. To enable the standard sidebar, add the following line to your pelicanconf.py:

SIDEBAR = "sidebar.html"

As always, you’re not limited to the standard sidebar implementation, and you can use your own custom sidebar template if you prefer:

SIDEBAR = "custom_sidebar_template_name_here.html"

Also, the relative width of the sidebar can now be changed using the SIDEBAR_SIZE setting. By default, if the sidebar is enabled it spans 3 columns in the responsive Bootstrap grid. The main content column takes up the rest. The value of SIDEBAR_SIZE can be set higher than 3 (but less than 12) to make the sidebar wider and content column narrower. For example, to make the sidebar and content columns the same size, set SIDEBAR_SIZE to 6 in your pelicanconf.py:

SIDEBAR_SIZE = 6

Custom CSS and Javascript

The STYLESHEET_FILES setting replaces STYLESHEETS.

The MAIN_LOCAL_STYLESHEET setting has been renamed to BOOTSTRAP_STYLESHEET. This option allows an alternative (customised) Bootstrap CSS file to be used in stead of the standard one.

As before, if BOOTSTRAP_STYLESHEET is not set, the standard Bootstrap CSS file is used via a CDN.

The above settings represent the primary means of incorporating custom CSS files, but in version 2, there are now even more options when it comes to configuring CSS and JavaScript files.

The new STYLESHEET_URLS setting allows additional style sheets to be linked in from arbitrary URLS (e.g. from a CDN). Once possible use case for this setting is to incorporate the Bootstrap theme CSS file from CDN, like so:

STYLESHEET_URLS = ("https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap-theme.min.css", )

Similarly, JAVASCRIPT_URLS and JAVASCRIPT_FILES settings have been added to make it easier to incorporate custom, site-wide JavaScript files.

For those who, for whatever reason, might not want to use any CDN files, it’s now possible to configure VoidyBootstrap without any dependencies on external CDNs. Doing this requires using the new SKIP_DEFAULT_CSS and SKIP_DEFAULT_JS settings, and then manually configuring all the theme’s dependencies (Bootstrap CSS and JavaScript, Font Awesome, jQuery) using the above settings:

SKIP_DEFAULT_CSS = True
SKIP_DEFAULT_JS = True
BOOTSTRAP_STYLESHEET = "bootstrap.min.css"
STYLESHEET_FILES = ("font-awesome.min.css", "pygment.css",)
JAVASCRIPT_FILES = ("jquery.min.js", "bootstrap.min.js",)

Note that if you go this way, it’s entirely up to you to download all those files and put them in the correct directories. (The theme is designed to use CDNs by default, and does not ship with any vendor files.)

Optional Templates removed

A core idea behind VoidyBootstrap is that most “blog” type websites have certain similarities in layout and overall structures, and therefore there are certain semantic content blocks that are likely to appear in most blogs (e.g. navigation, footer, author bio, etc).

Wherever practical, VoidyBootstrap will leave these content blocks blank and provide ways to theme users to add their own content. Or alternatively, provide a default implementation, but let theme users override this with their own, if they prefer (for example, like with the sidebar, as discussed above).

Version 1 of VoidyBootstrap had two ways to customise these “content blocks”: “Optional Templates” and “Custom Includes”.

Optional Templates have now been removed in favour of Custom Includes. Everything that was possible with Optional Templates in earlier versions can now be accomplished with Custom Includes.

Optional Templates seemed like a good idea at the time, but in practice I found them unwieldy. And it seems kind of redundant — not to mention potentially confusing to new users — to have two arbitrarily different mechanisms for essentially accomplishing the same thing. So Optional Templates are gone (vestiges may remain in the code, but these will eventually be removed — do not rely on them), and new Custom Includes have been added in their place (see below).

New Custom Includes

“Custom Includes”, for want of a better term, is a method VoidyBootstrap provides for customising content blocks. Custom Includes are settings beginning with the prefix CUSTOM_. They are all optional, but if used, must be set in pelicanconf.py to the filename of a Jinja template filename (or a list/tuple of filenames for some) which will then be added to the appropriate place in the website layout (via a Jinja include directive).

Lots of stuff has been changed when it comes to “Custom Includes”.

Some configuration setting were renamed, some were removed, and a whole bunch were added.

With the new “Custom Includes” settings, there is now far more fine grained control when it comes to adding blocks of custom content.

In many cases, it’s now possible to provided different templates for different page types (INDEX, ARTICLE, PAGE, CATEGORY, TAG, etc).

There are options for adding content outside of the main Bootstrap container used to wrap the 2 columns (CUSTOM_HEADER_*).

There are options for adding content inside the main Bootstrap container but above/below the columns, so in 2 column mode content will span above or below the 2 columns (CUSTOM_CONTAINER_TOP_* and CUSTOM_CONTENT_BOTTOM_*).

The settings CUSTOM_CONTENT_TOP_* and CUSTOM_CONTENT_BOTTOM_* are now the ones used for adding arbitrary content to the top or bottom of the content column.

Note that some of the new Custom Includes settings are lists: CUSTOM_SITE_HEADERS, CUSTOM_ARTICLE_HEADERS, CUSTOM_INDEX_ARTICLE_HEADERS, CUSTOM_ARTICLE_FOOTERS and CUSTOM_PAGE_FOOTERS. I realise this might be potentially confusing to new users of the theme. But I found with these settings in particular, it was often useful to be able to specify multiple templates. It makes it easier to split template fragments into smaller chunks, which could then be reused (DRY and all that).

For more details about the new Custom Includes see the “Custom Includes” section in the README, as well as the changelog.

Conclusion

The above is by no means a complete list of changes. Just the main ones (the ones most likely to affect anyone attempting to upgrade their installation from an earlier version). There have been other bits and pieces added or changed along the way. I think everything that deserves documenting has been documented in both the changelog and the README, although I might have possibly missed something.

If you do find anything I might have missed, let me know in the comments.

Smooth scrolling using jQuery

2015-05-29T00:00:00+10:00

Just leaving this here so I don’t have to keep looking it up…

Smooth scrolling is a common embellishment for web pages, and it’s possible to implement with surprisingly little code thanks to jQuery.

Creating a link that returns to the top of the current page can be done using an anchor with a URL fragment of “top”:

<a href="#top">Return to top</a>

Thanks to jQuery’s equivalents of CSS attribute selectors, we can easily add a click handler to all anchor elements on the page that implements smooth scrolling using the jQuery animate() function:

$("a[href='#top']").click(function() {
    $("html, body").animate({ scrollTop: 0 }, "slow");
    return false;
});

Floating scroll to top button

A more advanced embellishment is to add a floating button (often fixed in the bottom right corner of the page) that will return to the top. To implement this, first we need an appropriate anchor element somewhere on the web page. Here’s an example using Font Awesome for the up arrow icon:

<a href="#top" class="btn-scroll-to-top" title="Top of page">
  <i class="fa fa-arrow-circle-up fa-fw fa-lg"></i>
</a>

Then we style it appropriately with CSS and fix it’s position:

.btn-scroll-to-top {
    background-color: transparent;
    display: none;
    position: fixed;
    bottom: 20px;
    right: 20px;
    padding: 5px;
    font-size: 36px;
    font-weight: bold;
    text-align: center;
    text-decoration: none;
}
.btn-scroll-to-top:hover {
    text-decoration: none;
}

That by itself should do the trick (if we were to omit the display: none line). But we can take things a step further, and make the button fade in once the user starts to scroll down the page. Putting it all together with the smooth scrolling, the JavaScript would look something like this:

$(document).ready(function() {
    $("a[href='#top']").click(function() {
        $("html, body").animate({ scrollTop: 0 }, "slow");
        return false;
    });

    $(window).scroll(function(){
         if ($(this).scrollTop() > 200) {
             $('.btn-scroll-to-top').fadeIn();
         } else {
             $('.btn-scroll-to-top').fadeOut();
         }
    });
});

Here we’re using the jQuery scrollTop() function to determine how far the page has been scrolled, and toggling the button visibility as appropriate.

Laggy Lollipop: How to clear the cache partition on an Android Nexus 7

2015-03-17T00:00:00+11:00

I usually approach any software update with a sense of mixed feeling. Especially operating system updates. The promise of new features or better performance is offset by the trepidation that the new version might be substantially worse than what came before. (Yes, I’m looking at you Windows 8!)

My fears were realised when my Nexus 7 tablet, which had been happily running KitKat, received the Android 5.0 “Lollipop” update.

Things were definitely not right after that. My Nexus became slow. Really, incredibly slow and sluggish and laggy. With frequent ANR dialogs.

After the Lollipop update, my Nexus 7 was so sluggish it was close to unusable. After some googling it was clear I wasn’t the only one experiencing this problem.

I tried clearing the cache partition, and found that this helped improve performance somewhat. My Nexus is still performing much worse under Lollipop, but it has at least gone from “unusable” to “barely usable”.

The steps to do this are as follows.

How to wipe the cache partition

Start with the tablet turned off.

Press and hold the volume down and power buttons until a screen appears with a large arrow at the top (kind of pointing towards the power button at the top right corner of the device).
Press the volume down button a few times until an arrow with “Recovery” appears.
Press the power button to select Recovery Mode.

An Android logo with an exclamation mark should now appear (with “no command” text).

Hold down the power button, press and release volume up once, then release power.

You should now see a text based screen with a list of options.

Use the volume down button to move the selection to the “wipe cache partition” option. Double check to make sure you have selected “erase or wipe cache partition”. DO NOT select “wipe data/factory reset” by mistake!
Press the power button once to select the “wipe cache” option.

The process of clearing the cache should now begin, and will probably take a few seconds. Once it’s done, you should be back at the same menu.

If necessary use the volume buttons to highlight the “reboot system now” option.
Press the power button to reboot your device.

Once your Nexus finishes rebooting, it should hopefully be at least a little more responsive and usable. Whether the improvement lasts remains to be seen.

The Definition of Metadata

2015-03-03T00:00:00+11:00

Metadata used to be such an innocuous, mundane, technical term.

Until Big Government got hold of it.

And appropriated it for its own Orwellian purposes.

In technical circles, metadata is an abstract term. It means nothing in itself. What the word “metadata” refers to is entirely dependent on context.

If you’re talking about databases, metadata might information about the structure of the database — how the data is stored.

But you’re talking about photo files, it might be things like when the photo was taken, the location where the photo was taken, the device used to take the photo, the camera settings used to take the photo (exposure, white balance, shutter speed), etc.

It might be. But it depends. It’s an abstract term.

And it really, really depends on context.

Partly because it varies depending on context. The “metadata” of a photo sharing website is likely to be different to the “metadata” of an accounting application, for example.

It’s kind of like the word “religion”. It means different things, depending on which religion — or groups of religions — you might be referring to.

You might use the word “religion” in an abstract way, perhaps if talking about a group of religions in a general sense. But the word “religion” in itself doesn’t provide specifics — it wouldn’t be particularly enlightening to proudly proclaim a new found faith by saying you have decided to “convert to religion”. Which religion? It doesn’t make sense to get into specifics without providing at least some context about which religion you may be referring to.

Similarly, in the context of mass surveillance, talking about metadata without specifying the exact metadata doesn’t make much sense.

Yet, this is what the proposed mandatory metadata retention laws are doing — the legislation doesn’t even define the “metadata” that service providers are supposed to retain.

This is like saying religion is important and we must believe — without specifying which religion, or what we are to believe.

The Australian Government’s proposed data retention legislation takes its legal cues from The Castle — it’s all about “the vibe” of the metadata thing. The government is arguing these data retention laws are really important for national security, but apparently not important enough for specifics.

Which is kind of like arguing that all monotheistic Abrahamic faiths are basically the same and any distinction is not worth bothering with. Sure, you could try and argue that, but you’re likely to get some pretty strong push back in certain religious circles. I’d be willing to bet a lifetime supply of contraception on that.

With metadata as with religion, the devil is in the details.

And much like religion, the very notion of mass surveillance by the state requires a huge leap of faith.

Faith that data retention will be effect in catching criminals and preventing terrorism (despite evidence that it’s not).

Faith that the government and its agencies will not abuse the data it collects on its citizens (other governments have).

Faith that the metadata will be stored securely and won’t be hacked (unlikely/impossible).

Faith that it will not stifle freedom of speech and freedom of the press (it will by definition).

Faith despite evidence nothing the government says about data retention is true.

And it’s really hard to have faith in data retention when the argument for it has been so poorly articulated. The subtleties of metadata as an abstract term were obviously lost on Attorney General George Brandis when he tried to explain it in that now infamous interview on Sky News,. It’s hard to have faith in our elected leaders when they are making draconian laws about technology they clearly don’t understand.

After Brandis’s trainwreck of an explanation, we now have politicians glibly reciting the minimal route definition of metadata — that it’s “data about data”. While that might be the technical definition, it doesn’t really apply in the context of mandatory data retention. Like many terms used by politicians, “metadata” has been appropriated as a weasel word designed to deceive.

The term “metadata” is now part of the government’s attempt to make data retention seem benign — by making it sound technical; by implying metadata is somehow not actually data; by downplaying the privacy implications while exaggerating the distinction between “data” and “metadata”.

Anachronistic analogies describing metadata as “the material on the front of the envelope, not the contents inside the envelope” are misleading and inaccurate.

Metadata is a lot more than the address on the front of an envelope. Metadata is more like everything on the front and back of the envelope — including addresses, stamp, postmark and fingerprints — along with details of the pen used to write the address, where you bought the pen, where you bought the stamp, where you were when you wrote the address, who you were with, what you were wearing, what you had for breakfast that day, the date you posted the envelope, the postbox location where you posted the envelope, along with all your movements on the day you posted that envelope.

Ultimately, “metadata” is vague and abstract enough that in the hands of politicians it can mean whatever they want it to mean. This makes the technical definition irrelevant — any information that’s collected about us and our internet usage and communication is data. Don’t be fooled by false distinctions and technical connotations. The distinction between “metadata” and “data” is an arbitrary one, and in this case a misleading one. It’s all data, and it’s all private — even if it’s not considered equally private.

Sure, content of communication is considered more “private” than the “metadata” around that content (the contents of an email is considered very private, the time it was sent less so, for example).

But any consideration of the privacy of metadata is misleading if we only look at a single communication in isolation. Keep in mind that data retention is about our communication patterns in aggregate. It’s not just one envelope, it’s all the envelopes, and a whole lot more besides. Data retention is about gathering a large amount of data about all our digital communication and online activities, so that things can be inferred from sequences and patterns. Information that’s innocuous in isolation can become ominous in aggregate. Knowing an individual called a local doctor’s clinic might not reveal much. But knowing that call was followed by a numerous calls to abortion clinics possibly does.

The problem is with enough data, everything looks ominous in aggregate. Patterns truly innocent but seemingly nefarious will inevitably emerge due to randomness. The web browsing history of a chemistry student might, at times, look very similar to the web browsing history of a potential bomb maker. Journalists, activists, whistleblowers and researchers often have legitimate reasons for visiting dubious corners of the internet and perhaps communicating with individuals of dubious character — and doing so in private. But metadata by definition means no context or nuance. Metadata means everyone looks guilty. Metadata is an overbearing and constantly critical parent, always assuming the worst.

Just like we leave our scent behind as we move through the real world, in a constantly connected online world, we inevitably leave traces of our activities behind. Whether it’s an entry in a log file on an obscure server that will never examined, or an entry in a massive social media database that will be extensively data mined, it will still be there. Even if we don’t realise. We almost certainly cannot realise just what our digital metadata trail might imply. Things we would consider completely innocuous can appear suspicious taken out of context.

Like a sniffer dog tracking a scent, the government wants to be able to sniff out our online activities based on our digital data trail. But unlike a sniffer dog, the government has no leash, and no idea what it’s doing or what it’s looking for. Data retention gives government authorities the ability to look at patterns in our internet usage and communication, but no way to determine whether those patterns mean anything significant, or whether they’re just statistical noise. This means that when it comes to data retention, the government is a rabid, anxious, agitated beast ready to tear the throat out of anyone that smells a bit funny.

Data retention gives the government and its agencies too much power over us with too little oversight, for very little gain.

It’s also given “metadata” a new meaning: “metadata” is now a euphemism for government sanctioned spying.

Metadata means the government spying on all of us — but mostly the innocent — without a warrant.

Metadata means many false positives, which in turn means innocent people are be more likely to be charged and possibly convicted of things they didn’t do.

It means our phones become tracking devices for the government.

It means ISPs become agents of the state, gathering information about our online activities.

It means the government will know who we communicate with, and when, and how often.

It means whistleblowers will find it much harder to expose government and corporate corruption.

It means hackers will have an irresistible honeypot of data they will try and get their hands on.

It means the risk of identity theft will increase.

It means we have a government trying to implementing policy they don’t understand and can barely articulate.

It means wasting millions of taxpayer dollars on something that is known to be ineffective.

It means we get poorly drafted national security legislation based on fear rather than sensible, evidence-based policy.

It means online anonymity will be impossible.

It means invasion of our privacy.

It means undermining Australia’s democracy.

Misleading metaphors about envelopes don’t even begin to explain the deleterious effect mandatory data retention will have on our democracy.

Data retention is to democracy what a gatecrashing Satanist noisily sacrificing a goat is to a quiet christening— a disturbing and completely unwelcome intrusion.

Mandatory data retention: futile and dangerous

2015-02-28T00:00:00+11:00

They say the definition of madness is doing the same thing over and over again and expecting different results. So what, then, are we to make of the Australian Government’s fervour for data retention? Is it not the government taking something other countries have done and found to be ineffective, doing it anyway, yet expecting different results?

Mandatory data retention is in the news again in Australia, as an incredibly unpopular and incompetent Prime Minister scrambles back to his comfort zone of national security, and attempts to scare us into forgetting about his failings as a leader. So once again, Tony Abbott and his ministers are in the media saying silly things about technology they don’t understand, in an attempt to sell us the complete falsehood that data retention is required to fight crime and prevent terrorism.

Which is not to say terrorism is not a problem. It might not be as big a threat as the government and media make it out to be, but it is a threat. Which is why it requires sensible, evidence based approaches. As opposed to data retention, which represents an anecdotal, fear based approach.

Australia is not the first country to try mass surveillance of its citizens in an attempt to combat crime. Others countries have had data retention in place for a number of years already, which means we can learn from their experience. The evidence is fairly damning. The international experience shows quite conclusively that data retention doesn’t work. Not only is it completely ineffective, but it’s incredibly expensive, undemocratic, and open to abuse. It’s for those reasons many countries that have adopted data retention measures are now abolishing them.

But Australian authorities are determined to learn nothing from this, and are persisting in their misguided quest to spy on all of us.

While there’s been a backlash on social media to the data retention proposal, out in the real world people don’t really care, unfortunately.

For a country that’s somewhat cynical towards its politicians, we’re still remarkably willing to give government the benefit of the doubt. We shouldn’t.

Perhaps part of the apathy also comes from living in a world where organisations collecting data about us has become somewhat normalised. Whether it’s government or business, we know our actions are being tracked and stored. We know that social media companies track and store a huge amount of data about our online activities. We know that retail companies track our shopping habits through reward card programs.

But it’s worth noting there’s a big difference in the potential downside of a private company collecting data on users, and a government collecting data on its citizens.

For starters we have a choice whether to use the services of a private company, so we can always opt out. But if we do decided to “opt in” to a business and use its services, it becomes a value proposition. We accept that a social media website will track our usage, in exchange for the (free) service that social media website provides. We accept the retailer tracking our purchases, in exchange for whatever perceived benefit we may get (whether real or illusory) from the rewards program. If we decide the value provided by the service is no longer worthwhile, or the company displeases us enough in some way, we have the option of no longer doing business with them.

And usually the worst thing a private company can (legally) do is try to sell you something. Or sell data about you to some other company that tries to sell you something.

In contrast, the worst thing a government can do is take all your money and put you in jail for something you didn’t do — and maybe even execute you.

While the worst thing that can happen when Google’s advertising algorithm goes wrong is you will see an ad that has zero relevance to you. The worst thing that can happen if the government’s data retention goes wrong is you end up in jail for the rest of your life even if you didn’t do anything wrong. Or hackers get hold of your metadata and use it to steal your identity. Or you end up stalked by horny ASIO agents who have been illicitly reading your emails and looking at your private photos in the cloud. The possibilities for abuse are endless, but the benefits nonexistent.

There is no opt out from government mass surveillance, and no value proposition. What do we as citizens get out of it? Apart from a completely false sense of security? Apart from the illusion of government doing something about crime? Data retention has never worked to prevent crime or terrorism anywhere in the world where it’s been tried, so any perceived value is a mirage.

The benefits of data retention are illusory, but the downside is huge, especially considering the false positives problem. Terrorism is subject to the same principles of randomness and probability as everything else, and the false positives problem is the statistical reason data retention is ineffective in catching or preventing criminals.

With any kind of data analysis, the larger and broader the dataset, the more statistical noise, and the harder it is to analyse the data to find rare patterns or occurrences. When you have a large untargeted dataset, of the sort generated by mandatory data retention, meaningless patterns will occur out of sheer randomness. Just like when tossing a coin, it’s not unusual to get series of heads or tails in a row, but those consecutive heads or tails runs are just random noise and don’t tell us anything about the long term probabilities of the coin toss. This kind of randomness makes using a huge collection of data to find a very small number of criminals or terrorists pretty much impossible. Of particular concern is that any kind of analysis will generate a huge number of false positives — totally innocent people the algorithm has flagged due to randomness. No matter how good the algorithms may be, not matter how advanced the analysis, there will still be a huge number of false positives, because that’s how the probabilities play out.

Given that the worst thing a government can do if it’s wrong about an individual is take away everything they own and deprive them of their freedom for the rest of their life, mandatory data retention should be of concern to everyone. Especially innocent, law-abiding citizens. Those thinking they have nothing to hide because they haven’t done anything wrong are in for a rude awakening if they’re unlucky enough to end up being one of false positives.

Mandatory data retention will increase the chances of innocent people being convicted of crimes they didn’t commit, all the while diverting police resources from pursuing actual guilty criminals.

Any kind of analysis on the metadata database will flag so many genuinely innocent people as potential criminals worthy of investigation, it will be impossible to sift through them all to find the truly guilty. Police and intelligence resources will be wasted on wild metadata driven goose chases, trying to establish that innocent people are actually innocent. Or worse, prosecuting the innocent.

Authorities already have considerable powers for targeted data collection on specific suspects. It is extreme overkill, not to mention extremely expensive and counterproductive, to use mass surveillance to collect data on everyone. Especially the innocent.

Who cares about data retention anyway?

2015-02-23T00:00:00+11:00

There’s a whole world of apathy out there…

“The Australian people are being sleep walked into a system the Attorney-General cannot even articulate.”

—Police insider speaking to ABC Radio

Over the weekend I caught up with a couple friends I hadn’t seen for a while. Friends outside of the IT industry.

At one point, when the conversation turned towards politics, I asked if they had heard about the government’s proposed mandatory data retention plan. Neither had.

“Don’t get him started,” said my girlfriend, as I launched into a variation of a rant she’d probably heard before.

I’m not sure if I should give her points for putting up with my political/geeky rants, or question our relationship given her disturbing lack of feels about privacy and democratic freedoms? (Maybe she’s an ASIO spy?!)

These two friends were reasonably active on Facebook (not so much Twitter), and lately the only thing I’ve been doing on Facebook (and Twitter) is sharing articles about data retention (I am really fun on social media).

Which raises the question of whether these (supposed) friends were ignoring my Facebook posts from lack of interest, or if Facebook’s algorithm didn’t bother showing my activity in their feeds?

It would not surprise me if it was the former. Not everyone amongst my friends and family shares my zeal for technology and politics issues. Actually, few do, even those more technically inclined.

My brother, for one, is usually quite blunt about his lack of interest in most things I deem worthy of sharing on Facebook. We might share DNA, but he shares little of my computer geek interests. He has no qualms about telling me that even though he sees most of my status updates in his Facebook feed, he ignores almost all of them because they are of no interest to him. And because any articles I share usually require too much reading.

Surrounded by such indifference, I have resigned myself to the fact that I am merely shouting into the void on social media (and this blog).

Still, while not necessarily surprising, it’s still somewhat disconcerting that others can be so oblivious to an issue that has had reasonable media coverage, and quite a lot of traction on social media.

But by its personalised nature, everyone’s experience on social media will reflect their interests. Especially in the echo chamber of Twitter.

While my Twitter feed is a constant stream of anti-data-retention vitriol, that’s unlikely to be everyone else’s experience.

Even amongst “friends” on Facebook, an item shared is not necessarily an item that anyone else payed any attention to whatsoever. Especially if it was about something as “niche” as data retention.

While I, and many others, obviously have some strong opinions about data retention and privacy issues, and have expended considerable effort trying to spread the word online, that doesn’t mean it has grabbed the attention of the masses.

Things are very different in the real world, outside of Twitter and the IT industry and geeky social circles. Sometimes social media makes it easy to forget that.

I’d bet most people are largely indifferent to the issue of mandatory data retention.

Out in the suburban mainstream, many people haven’t heard of data retention. And even if they have, they don’t really care. It doesn’t seem like something that should matter to them, with their McMansions and their weekend sportsball. Even though it does.

Which once again confirms my belief that social media might have its uses, but it’s not a particularly effective tool for activism. Few people have changed their minds because of a pithy tweet or dank meme. Social media tends towards preaching to the converted. Communicating to the apathetic masses through social media doesn’t work for seemingly dry tech issues like data retention, even though the privacy implications are wide ranging and will affect everyone using the internet.

In my previous post I pondered politicians not taking social media criticism seriously. This is another reason why they won’t. I suspect they realise the repercussions of social media are relatively benign. At most, a social media backlash might get generate a few fleeting minutes of coverage within the 24 hour news cycle, in between the real money shots of news — celebrity gossip and sport.

This is why, despite all the angry tweets about data retention, despite the shared links to articles pointing out the flaws in the concept, our politicians will ultimately get their way. Our politicians will succeed at undermining our democracy and eroding our privacy.

Because most people just don’t care.

And those that do are just shouting into the void.

Electronic Graffiti

2015-02-18T00:00:00+11:00

Australian politics is weird at the moment. More so than usual.

We have a conservative Prime Minister so incompetent and unpopular that even conservatives are having some pretty intense buyers’ remorse.

His own backbenchers even tried to get rid of him. But he survived, so the country is still stuck with Tony Abbott as PM. For now.

Tony Abbott survives to gaffe another day, and if past performance is any indicator of future performance, there will be more gaffes to come. Many, many more.

Abbott just stumbles from one gaffe to another. We barely have time to laugh, scoff, scorn, shake our heads and wonder in disbelief at what Abbott was thinking when he said that — before he goes and says or does something else even more astonishingly stupid.

Or one of his ministers obliges.

When Attorney General George Brandis hilariously bungled an attempt to explain metadata on Sky News, it quite rightly made him an object of ridicule. But not for long enough, in my opinion. I felt this particular fuckup was so bad yet so easily preventable (do some research Brandis, it’s your job!), that the public deserved to dine out on it for much longer. Despite the ridicule he got, Brandis deserved a lot more, since he clearly did not have any understanding of laws he was proposing. It got a bit of media attention, then the media focus quickly moved on to other senior government ministers embarrassing themselves, so I think we got a bit short changed. Brandis’s self-inflicted metadata mess was soon overshadowed by other government ministers saying stupid shit — Treasurer Joe Hockey saying poor people don’t drive cars, and Senator Eric Abetz claiming abortion causes breast cancer.

Stupidity must be contagious in the Liberal Party.

But seems even the Liberal Party must draw a line somewhere. And looks like Abbott crossed that line by giving a knighthood to British royalty. On Australia Day.

Yes, the Prime Minister of Australia bestowed a knighthood on Prince Philip, the Queen’s husband. On a day when the country celebrates the achievements of great Australians, Tony Abbott decided to honour British royalty instead.

Obviously, a lot of people thought this was really stupid. It was another episode in his sitcom of a government; another moment when the nation collectively asked: “What the hell was Abbott thinking? Didn’t he run that idea by anyone first? Doesn’t he have advisers?”

Quite predictably, social media had a field day with this. Especially on Twitter.

But Abbott dismissed the outpouring of scorn on social media as electronic graffiti. His exact words were:

Social media is kind of like electronic graffiti and I think that in the media, you make a big mistake to pay too much attention to social media.

You wouldn’t report what’s sprayed up on the walls of buildings and look, as I said, social media has its place, but it’s anonymous. It’s often very abusive and, in a sense, it has about as much authority and credibility as graffiti that happens to be put forward by means of IT.

Predictably, this didn’t help his cause much, and it definitely didn’t make him seem any less out of touch.

But while the “electronic graffiti” line will no doubt live on as an ironic Twitter hashtag for a while yet, once again mainstream media coverage (and derision) was disappointingly brief, as leadership speculation and the subsequent spill motion became the story.

So I want to pause for a moment, and ponder the electronic graffiti comment a bit.

Sure, it’s a another sign that Tony Abbott knows little about technology, and certainly doesn’t “get” social media. Dismissing what has become a mainstream communication and entertainment medium in this way is just weird.

But treating this as another example of Abbott being out of touch and technologically illiterate is far too simplistic. Maybe even far too generous. I think it goes far deeper than his apathy — and sometimes outright disdain — for science and technology.

Abbott’s “electronic graffiti” comment is nothing less than a sign of his utter contempt for voters combined with an unbounded ego. The subtext behind it isn’t just that things people say on social media don’t matter — it’s that things people say in general don’t matter. Abbott simply does not care about other people, their opinions, or criticisms. His overblown ego doesn’t allow it. Hence his many silly “captain’s picks”, made with little consultation, and probably with the firm belief he was making great decisions.

Tony Abbott has a weirdly inflated view of his own superiority and infallibility. Even his own backbenchers are not worth listening to, it seems. The recent spill motion was initiated by backbenchers who felt their views and need were being ignored. If Abbott doesn’t care what his ministers or backbenchers think, he sure as hell won’t care what randoms on Twitter think of him.

Another manifestation of this mindset was demonstrated during his National Press Club speech, when Abbott insulted voters and disparaged democracy by saying people elect Labor governments in a fit of absent-mindedness. Tact and subtlety are not exactly hallmarks of Abbott’s communication skills. Neither is any awareness of irony — someone as famously gaffe prone as Abbott really isn’t in any position to be calling anyone else “absent minded”, and certainly not millions of voters!

Of course, an egotistical and arrogant politician is hardly a rarity. It’s more like a prerequisite for the job. But it helps if there is a vision and intelligence — there’s little evidence Abbott possesses either — to back up the attitude. Abbott’s confidence is based purely on his own delusions.

All of this has some potentially profound implications for those contemplating any kind of political activism.

Some people use social media with an determined earnestness, labouring under a false belief that their social media sharing and posting might make a difference, somehow. That their Facebook posts, tweets, and blog posts actually matter.

I doubt it. The idea that social media makes a difference is just as delusional as Abbott’s belief in his own superiority.

I’ve always been sceptical of the effectiveness of online activism of any kind. Whether it’s the slacktivism of online petitions, the clicktivism of groups like GetUp, or the various combinations of blog posts, memes, Facebook status updates and tweets — mostly it’s just shouting into the digital void.

Proof of this abundant, if we look at the many cases where online activism failed. Especially in technology-related issues, which are more likely to provoke online angst.

Take the National Broadband Network, for example. Few people within the IT industry thought the Coalition’s alternative to the NBN was a good idea. Pretty much nobody on Twitter thought it was a good idea. But online petitions and widespread derision on Twitter didn’t stop the Coalition from destroying the NBN. What we’re getting now cannot even be accurately as either “national” or “broadband”, and it’s barely even a “network”. Labor’s vision for universal fibre is pretty much dead. All the online outrage and well reasoned articles and blog posts didn’t save the NBN.

Then there’s the new National Security laws, many of which are designed to restrict online freedoms. The first two tranches of National Security laws were unnecessary, un-democratic, and deeply flawed. Reaction on social media (and also mainstream media outside of Murdoch publications) was mostly negative. Yet, the legislation passed. And there’s more to come with the third tranche of laws — for mandatory data retention — still before the parliament.

The battle over mandatory data retention is still being fought, but like the other battles between the technology industry and the government, it’s one were the government will probably win. The government will win because people outside the tech industry don’t care, and those within it who do care are wasting time online instead of taking the fight to politicians offline. Despite widespread criticism on social media and opposition from the telecommunications industry, mandatory data retention is likely to be implemented. The government will succeed at forcing the telecommunications industry to implement ineffective, expensive and misguided policy.

Of course, few would have realistically expected much from this government in terms of science and technology. Though many might be surprised by the level of hostility towards issues in these areas — not to mention abject incompetence. From broadband to climate change, this LNP government seemingly prides itself on being behind the times. Australia currently has a government suspicious of any technology that’s not directly coal-related.

We can certainly forget about any prospect of sensible technology policy from this government. Not while we have a Prime Minister who thinks the internet is “essentially a video entertainment system”.

The best we can hope to achieve is to stop these idiots messing things up too badly — stop them implementing bad laws and bad policy based on outdated beliefs and flawed ideology. Our chances don’t look good based on past performance. Especially if we’re going to rely on social media as our main strategy. (Thank God for a hostile Senate!)

Tony Abbott’s view of social media as “electronic graffiti” is further evidence of the impotence of social media in influencing the political process. I suspect it’s more than just another of his weird-ass, brain-fart soundbites. In this case, it’s a reflection of his true beliefs — and from his position of power and privilege, those beliefs become self fulfilling prophecy. By believing social media is not worthy of his attention, Abbott will happily ignore it, as will his ministers and his government.

Another reason I think social media matters little to politicians is that they’ve made little attempt to regulate it. Politicians love to regulate stuff. It’s kind of why they exist. They regulate things even if there’s no sensible reason to regulate it (the data retention thing is an example of that). I suspect the reason that they’ve largely left social media alone is partly because they don’t really understand it (even though they’re spending a lot of money monitoring it), but mostly because they realise it’s an excellent (and low cost) marketing tool for their own purposes. Which is essentially the whole point of social media. Some users might think they’re being subversive by posting political content, but it’s all part of the plan. Social media is mostly a platform for advertising and marketing. It is just entertainment for the masses. In the bread and circuses of politics, social media is just another circus.

Maybe one day social media will have real influence in the political process, but I suspect that day is at least a generation away. Maybe when the current generation that has grown up with social media is in power, they will be more inclined to pay attention to the vitriolic rantings of the Twitter hoards. Although by then social media will probably have morphed into a different (more corporatised? more regulated?) beast than what it is today.

Or maybe social media will remain mostly a distraction for the masses, and mostly useless for political activism. While online activism may occasionally have a win with little issues, it rarely has much effect when it comes to big political issues — especially issues driven by corporate interests with lots of money on their side.

In a way, the current Coalition government is a case study in the ineffectiveness of online activism. Just about everything they’ve done — from border protection and asylum seekers to national security — has been met with widespread criticism on social media. As a government, they are incompetent and clueless, and about as unpopular as it’s possible to be in Australian politics. Yet, they’ve been remarkably successful at implementing bad laws and making bad decisions, particularly on issues affecting technology users and the technology industry, where the online backlash has been especially harsh.

Despite this, they won.

They won because governments don’t care about criticism on social media. They won because there aren’t enough votes or campaign donations in technology issues for it to matter that a bunch of nerds on Twitter are angry at them.

We’re going to need a lot more than dank memes to make a dent in the entrenched LNP apathy to technology issues.

By all means use social media to vent.

Just don’t do so thinking it will achieve anything. Because if past performance is any indicator, it won’t.

The Who Hacked Sony Story

2015-02-02T00:00:00+11:00

The recent Sony hack was notable for many reasons, not least of which is how devastatingly successful it was — hackers managed to get hold of huge amounts of sensitive data.

But one aspect that intrigued me in particular was the way it was reported, and how different angles of the story were emphasised (or de-emphasised) by different media outlets.

Obviously this story would be of some interest to those technically inclined, especially those working in areas of technology or cybersecurity. As such, there are some complex technical aspects, which might be of interest to a certain tech savvy demographic, but perhaps not so much to a more general mainstream audience. But while it’s one thing to tailor the amount technical detail in reporting to the level of your target audience, it’s another thing entirely to just regurgitate government propaganda without any attempt at providing context.

Yet, that increasingly became the nature of the reporting, especially in the weeks after the story first broke, as more details were made public. In a way, the more information that emerged, the more distorted that information became.

Pretty soon, the “official” narrative became that the North Korean government was responsible for hacking Sony. After that most major “mainstream” news sources just reported this as if it was a proven fact, without any attempt at context or nuance.

Anyone with just a passing interest in the story, who was getting their news only from popular mainstream news websites, would be left convinced that hacking Sony was all part of a cyberwar North Korea was waging on its perceived enemies in the West.

This is in stark contrast to the way this story was reported on technical news sites, and the way knowledgeable bloggers wrote about it.

The North Korea cyberwar narrative might make for a better story, and a more dramatic story, but it there’s a lot more going on than that.

Right from the start, many security experts considered it unlikely North Korea were the perpetrators. With these kinds of hacking attacks, attribution is notoriously difficult. So far, all the evidence linking it to North Korea is circumstantial at best. Sony is also a strange and rather low value target for a nation state. The case that Sony was targeted because of “The Interview” movie is weakened by the fact the attackers only mentioned the movie after the media started speculating about this possiblity. Surely these points are worth noting, if only in passing, in any objective reporting of the story? Even one aimed at a non-technical audience?

While it’s hardly surprising to see tech news reported badly or innacurately, it was still dissapointing — and disturbing — to see how pervasive the official government line became, that North Korea was responsible.

Especially considering the increasingly likely theory that the Sony hack was actually the work of disgruntled former employees, who may have planted malware software from the inside, stolen data, and handed over crucial details of Sony’s internal network to hackers.

I’m no security expert, but this seems like a far more likely scenario. The hackers were so successful that it seems inconceivable they could have achieved as much as they did without help from the inside. It’s astonishing that hackers might have managed to transfer so much data out Sony’s network without anyone noticing. If that is what happened, then Sony had some seriously inadequate network monitoring in place. Even so, isn’t it far more likely that some disgruntled employees copied huge amount of sensitive corporate data onto external USB drives, then just walked out the door with them?

Of course, the US government has officially blamed North Korea for the cyber attack on Sony. Which doesn’t mean it’s necessarily true. Sure, with all they spying the NSA is doing (along with other government agencies) it’s certainly likely that the US government knows a lot more than they are publicly revealing. But blaming North Korea is awfully convenient, considering we may never know for sure who is responsible.

After all, what’s more likely: That a secretive, poor, communist country with little internet infrastructure launched a phenomenally successful cyber attack against Sony because they were about to release a crappy Seth Rogen movie? Or that disgruntled employees planted malware, then handed over the details to some hacker group? My money is on the latter.

But most of the media went with the official government line.

So much for the internet setting truth free.

Anyone else remember when the internet first started to become mainstream? Remember the hype of the dot-com boom? We were entering a new age, we were told. Old media was now obsolete. The internet would obliterate all the old publishing empires, and bring in a glorios new age of self publishing. No more would media moguls and governments control the flow of information. Because information wanted to be free! The truth is out there!

Didn’t really turn out that way, did it? Maybe kinda sorta, if you squint real hard, and lower your expectations. There are still book publishers, record labels, newspaper companies. Sure, many of them struggling for profitability or relevance, but they’re still there. More importantly, a lot of the old media empires are still around, and still have substantial power and influence.

Even the advent of social media hasn’t really made it easier to find the facts. If anything, it’s made things worse. Social media provides just another outlet for governments and corporations to get their message out. We have access to far more media than ever before. But not necessarily more facts. It’s probably no coincidence that the more media there is, the more mis-information is out there. Who needs to censor the media, when you can just bury inconvenient truths in amongst a sea of mis-information and spin?

Which I think is the ultimate irony of the Sony hacking. Despite little evidence, large swathes of the media are happily reporting the official US government line that North Korea is responsible for the cyber attack on Sony.

North Korea, one of the last die hard vestiges of communism, is a country where few people have internet access, and the media is mostly government propaganda designed to keep its population misinformed.

By contrast, the supposedly democratic Western world enjoys a mind-boggling array of media options, from traditional news sources, to obscure news websites and blogs, to social media. Yet, ironically, this doesn’t always mean we are told the truth. Are the people in the western world any less mis-informed?

We might get more news, but it’s still filtered through layers of government spin, corporate PR, and news media bias. As much as the internet might hold potential to speak truch to power, it’s more often a vehicle for conspiracy theories, and outright mis-information. Not to mention cat pics.

The truth might be out there, online. But we usually have to wade through a big pile of shit to find it.

Hello 2015, and goodbye online freedom and privacy

2015-01-12T00:00:00+11:00

It’s that time, as the new year gets underway, and memories of New Year’s hangovers recede, where many a second-rate writer wastes many words attempting to predict what’s ahead for the new year.

Since describing my blogging attempts as “second-rate” would be exceedingly generous — and with slim chance my online ramblings will have anyone of sane disposition mistake me for a writer — I feel I have little credibility to lose by throwing my contribution into the mix.

In 2015, the disturbing trends of recent years will no doubt continue — mass surveillance by governments, erosion of digital rights, widespread invasion of privacy, stupid software patents, and Luddite legislators enacting stupid laws affecting the tech industry.

This year will be more of the same. It’s a war, against us, and our freedom. We, the users and developers of technology, will lose. Politicians and large corporations will win. Again.

This will likely be the year the Trans-Pacific Partnership will be finalised. The TPP means more power for large corporations, and less for consumers and governments. It means consumers will pay more for everything from movies to medication. It means governments will become more vulnerable to lawsuits from multinational corporations. It means countries will be forced to introduce draconian copyright provisions that criminalise infringement and significantly curb online freedom.

In the US, this will probably be the year that the battle for net neutrality will be lost, setting a disturbing precedent for the rest of the world. For those unsure about the implications of net neutrality, is was nicely summed up by John Oliver on Last Week Tonight as “cable company fuckery… because that is what it is.”

Is there anything more disturbing than the prospect of cable companies becoming more powerful?

2015 will be a year of even more mass surveillance by governments. In the US there’s already a staggering amount of mass surveillance being undertaken by the NSA and other government agencies, and this will no doubt keep increasing.

Other countries also have data retention regimes in place to routinely spy on their own citizens, and more are likely to join the mass surveillance circle jerk, despite little evidence it helps in combating crime or preventing terrorism.

Never one to let evidence get in the way of bad policy, in 2015 the Australian government will introduce its own flawed mandatory data retention scheme, conveniently ignoring evidence showing similar schemes in other countries have failed to provide any benefit.

Last year, the Australian government successfully introduced legislation that undermined basic democratic rights of free speech, freedom of the press, freedom of association, and the right to be considered innocent until proven guilty. This year, they will continue their attack on democracy by eliminating our right to privacy through mandatory data retention. And possibly restrict our internet access by introducing a filtering scheme.

Throughout all this, I’m sure many technology literate people will respond with much anger and vitriol on social media, along with the occasional well reasoned blog post or article. But as in previous years, none of that which will achieve anything. We will lose. Because this is a war against us, but it’s not a war where facts or logic matter. It’s a war where money, spin, corporations, and political lobbying decide the outcomes.

In 2015, as in years before, the politicians and their corporate masters will get their way.

Happy new year!

Upgrading to pelican 3.5

2014-11-30T00:00:00+11:00

I recently updated my Pelican blog setup, going from Pelican 3.3 to Pelican 3.5. I skipped Pelican 3.4, mostly by circumstance rather than intention — by the time I got around to upgrading, Pelican 3.5 had been released!

Although I wasn’t in any great rush to upgrade anyway, as I was rather diligently procrastinating the task of reinstalling the Linux box I’d been using for my Pelican work, and this is something I wanted to do first. But by the time I had overcome my procrastination (not to mention general reluctance to mess with Things That Are Working), I’d discovered that Pelican 3.5 was out. So perhaps in a way my procrastination resulted in some fortuitous timing.

For the most part things went surprisingly smoothly.

Since I was essentially reinstalling (and reconfiguring) things from scratch on a new Linux installation (Mint 17), I decided to have a go at using Python 3, as up until now I’d only ever used Pelican with Python 2.7. I used the system’s Python 3.4 (and “pip3” via the system’s package manager).

Not being sure if virtualenv/virtualenvwrapper played nice with Python 3, I thought I’d try the native virtual environment support in Python 3: venv.

Here, I ran into a bit of a problem. When trying to create a new virtual environment I’d get an error like this:

Error: Command '['/home/rob/python/newenv/bin/python3.4', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

This appears to be caused by an issue in Python 3.4 (affecting Debian and derivatives, at least).

The solution here fixed the problem.

After that I was able to create a new virtual environment for Pelican with pyvenv:

mkdir venv
cd venv
pyvenv-3.4 pel
source pel/bin/activate

From this point on things were fairly straightforward. I didn’t encounter any issues with using Python 3 instead of Python 2 to install and configure Pelican:

pip3 install pelican markdown

The above command installed Pelican 3.5. With that done, I did a dummy run of pelican-quickstart, then compared the various generated files to that of version 3.3, to get an idea if any important differences needed to be addressed.

There wasn’t much. Here is a diff of the Makefiles:

1,2c1,2
< PY=python
< PELICAN=pelican
---
> PY?=python3
> PELICAN?=pelican
27a28,29
> GITHUB_PAGES_BRANCH=gh-pages
>
90c92
<     rsync -e "ssh -p $(SSH_PORT)" -P -rvz --delete $(OUTPUTDIR)/ $(SSH_USER)@$(SSH_HOST):$(SSH_TARGET_DIR) --cvs-exclude
---
>     rsync -e "ssh -p $(SSH_PORT)" -P -rvzc --delete $(OUTPUTDIR)/ $(SSH_USER)@$(SSH_HOST):$(SSH_TARGET_DIR) --cvs-exclude
99c101
<     s3cmd sync $(OUTPUTDIR)/ s3://$(S3_BUCKET) --acl-public --delete-removed
---
>     s3cmd sync $(OUTPUTDIR)/ s3://$(S3_BUCKET) --acl-public --delete-removed --guess-mime-type
105,106c107,108
<     ghp-import $(OUTPUTDIR)
<     git push origin gh-pages
---
>     ghp-import -m "Generate Pelican site" -b $(GITHUB_PAGES_BRANCH) $(OUTPUTDIR)
>     git push origin $(GITHUB_PAGES_BRANCH)

(Note that a Python 2 environment was used to generate the Pelican 3.3 files, while Python 3 was used for Pelican 3.5. I’m not sure if that’s significant, I don’t think it was.)

There appear to be only minor changes to the deployment targets. The only one of note being the addition of the -c option to the rsync command.

Here is a diff of pelicanconf.py:

8a9,10
> PATH = 'content'
>
11c13
< DEFAULT_LANG = u'en'
---
> DEFAULT_LANG = 'en'
16a19,20
> AUTHOR_FEED_ATOM = None
> AUTHOR_FEED_RSS = None

So pretty much the only change I had to make was disable author feeds, which I had no use for.

Both publishconf.py were identical.

The develop_server.sh script had many minor changes — almost all of them to the wording of text in echo statements. I just copied over my old version with the latest.

There was only one problem I ran into during the upgrade, and it’s actually a problem I first noticed after I upgraded to Pelican 3.3, which changed the way static files were handled and introduced the EXTRA_PATH_METADATA setting. I had a file with a HTML extension that I simply wanted copied into the root directory of my generated website, without being processed in any way. The file wasn’t actually a valid HTML file though. It was just an arbitrary text file, however it was generated by an external tool, so I couldn’t change the contents of the file, or its file extension. My initial attempt to configure this resulted in something like this in my pelicanconf.py:

STATIC_PATHS = ['images',
                'other/robots.txt',
                'other/favicon.ico',
                'other/arbitrary.html',
                ]

EXTRA_PATH_METADATA = {
    'other/robots.txt': {'path': 'robots.txt'},
    'other/favicon.ico': {'path': 'favicon.ico'},
    'other/arbitrary.html': {'path': 'arbitrary.html'},
    }

That worked for the first two files, which were copied across correctly to the root directory of the generated website. But the file with the HTML extension wasn’t copied, and Pelican output this error message:

ERROR: Skipping other/arbitrary.html: could not find information about 'title'

My initial attempts at finding a solution had been unsuccessful, so I had been ignoring this error for a while. But I eventually noticed this file not being copied to the right place was causing some issues, which motivated me to once again try and find a solution.

I came across some discussions of this problem in these two issues on GitHub.

Turns out that Pelican accepts HTML files as source input, just like Markdown and reStructuredText, which is probably why it was attempting to process my arbitrary (non) HTML input file, and complaining about a missing title. This is all well and good, but I can imagine many valid use cases where it might be desirable to have HTML content just copied over as part of site generation without any processing (like content generated by tools external to Pelican). Fortunately, Pelican provides a way to skip processing of all HTML files via the READERS setting:

READERS = {'html': None}

This solved the problem and got rid of that error message. After adding the above line to my pelicanconf.py all my external, non-generated files ended up where I wanted them in the output directory, and I finally had a “neat” build without any mysterious warning or errors.

Overall, the upgrade went fairly smoothly.

I didn’t encounter any backward compatibility issues with Pelican 3.5. Apart from the minor tweaks above to Makefile and pelicanconf.py, there were no significant changes required to either my configuration or (custom) theme.

I did, however, encounter a minor glitch, which I think had something to do with caching.

Pelican 3.5 features caching to speed up site generation, which is nice and usually works quite well. It doesn’t appear to be perfect, though. At one point, while messing around with theme modifications, I noticed that some parts of my website were not being generated properly. In particular, there were internal links that were pointing to wrong (non-existent) URLs. At first I thought I had introduced some weird theme issue, but that didn’t appear to be the case.

I deleted the cache directory. Then I did a make clean followed by a make html That fixed the problem — the regenerated website was fine. I’m therefore assuming it had something to do with the caching. I wasn’t able to reproduce the problem though, so for now it remains a vague observation that the caching can sometimes cause issues.

Metadata karma

2014-11-28T00:00:00+11:00

The government did a couple things I found really disturbing recently.

Well, actually, that seems like a ludicrous understatement given recent events in Australian politics.

Recently, everything this government has done has been disturbing on some level.

For the sake of brevity however, if nothing else, let’s consider just two for now: the mandatory data legislation introduced in recent months; and the prosecution of 21 year old student Freya Newman for uncovering evidence of a secret scholarship awarded to the daughter of the Prime Minister.

The latter has recently concluded, with Freya Newman being sentenced to a 2 year good behaviour bond earlier this week. The former will drag on into next year, the government’s attempt to ram the legislation through like the first two tranches of new national security laws having thankfully failed.

While not directly related, apart from the fact I consider both to be quite shitty things for a government to do, I think it’s interesting that the debates surrounding both played out around the same time.

They’re both issues that you can slice and dice in different ways, and look at from different angles. Generally, I suspect one’s interpretation of both to be colored by how much one trusts the motives of government in general, and this government in particular.

Personally, I have little benefit of the doubt left to give. I have little trust in any government generally, abject apprehension of our current one, and nothing but outrage and dismay at both the prospect of metadata retention and the way Freya Newman was treated.

On both issues I think government motives are far from pure.

For mine, there is little ambiguity: mandatory data retention is an invasion of privacy and represents massive government overreach; Freya Newman might have technically broken the law (possibly a stupid law), but she did nothing wrong, and the information she revealed was definitely in the public interest.

As Senator David Leyonhjelm says, “We should be watching the Government, not the other way around.”

Which is why Freya Newman’s actions were in the public interest. As voters, we have a right to know that a private design school — which stood to benefit from an Abbott government’s policies — had given Frances Abbott a secret scholarship. Not to mention the hypocrisy of the Prime Minister’s daughter given a free ride after all the “lifters and leaners” rhetoric by this government. Probably why the Prime Minister hadn’t publicly declared the scholarship. It was an embarrassing revelation for the Prime Minister.

But while the public has a right to know what our politicians are up to, supposedly on our behalf, the government has no right to invade the privacy of law abiding citizens. Spying on the innocent with mass surveillance is going to far. It will not make us safer from terrorist. But it will undermine our democracy, and our privacy.

Amongst other things, data retention is a deterrent to whistleblowers coming forward to expose government (or corporate) corruption. Which is another aspect the data retention laws have in common with Freya Newman’s prosecution — the intention of both is to discourage whistleblowers. Well, whether it’s an intention or a side effect is debatable, but it will be the result nonetheless. A result that’s beneficial to government, but detrimental to democracy.

Freya Newman was prosecuted to send a message to anyone else who might consider revealing information that proves inconvenient to the government. As her lenient sentence demonstrated, her “offence” was a minor one at most. What she did wouldn’t even have been illegal if it involved a public university, as opposed to a private school.

The authorities have discretion about what they do or don’t prosecute, and it’s likely Freya Newman was only prosecuted as punishment because she revealed information embarrassing to the current Prime Minister. It’s unlikely she would have been charged if the scholarship in question had involved someone less high profile than a sitting Prime Minister’s daughter. It was a vindictive use of the legal system to punish a past whisteblower and discourage future ones.

With the prosecution of Freya Newman, this government has shown it doesn’t take kindly to whistleblowers, and is willing to abuse the legal system to punish them. Even if they’re just a 21 year old university student. This is a reason in itself to be suspicious of government motives when it comes to data retention. While the government might say it’s all in the interests of national security, their real agenda likely involves some combination of whistleblowers, activists, and illegal downloaders.

However, there might actually be a small upside to all this. The government has shown itself to be way out of its depths on the technical issues surrounding metadata retention. Or even definitions of metadata.

Politicians also have a remarkable ability to either ignore or underestimate side effects of their laws, in their misguided belief there is no problem that can’t be legislated away.

These two factors could combine to produce some entertainingly ironic results.

While metadata retention will likely result in less whistleblowers coming forward, it creates the all too likely possibility that the retained metadata will be hacked and leaked. Any hacked metadata will likely contain metadata on politician, their families, business associates, mistresses, etc. This leaked metadata could be mined to expose information embarrassing or damaging to the government. So when one door closes, another opens.

Because if and when mandatory data retention is introduced — and it probably will be, eventually — I can’t imagine any scenario where ISPs are forced to store so much data unnecessary to their core business, and manage to do so securely.

The metadata will inevitably be hacked. It will be hacked because data retention will be really expensive, and IT security is complicated and difficult at the best of times. Already there is squabbling about who will pay for it. Which means it’s likely to be us, the consumers. And when time or money constraints impact any IT project, security is the first thing to suffer.

So the one possible bright side from the government introducing a misguided mandatory data retention regime, possibly in an attempt to discourage whistleblowers revealing government secrets, is that it will lead to hackers getting at the metadata of politicians and revealing government secrets.

The results should be entertaining, and the schadenfreude will be exquisite, as hopefully those politicians who voted for stupid metadata laws will have their careers embarrassingly destroyed by an unnecessary metadata regime they introduced.

It will be karma through metadata. Hopefully.

We all have something to hide

2014-11-25T00:00:00+11:00

The Australian government recently introduced mandatory data retention legislation, and in the debate that followed, I heard a radio talkback host admit he wasn’t overly concerned by the prospect, on the grounds that if you don’t do anything wrong you have nothing to hide.

He couldn’t be more wrong.

Thinking “I have nothing to hide” is naive when it comes to mandatory data retention, and represents a failure to fully understand the implications.

Data retention is usually thought of as a technical issue, but that is a far too simplistic way of looking at it. Mandatory data retention has profound implications for privacy, the media, and basic democratic rights.

It’s not an abstract thing — data retention is every much an invasion of privacy as would be Tony Abbott standing behind you and watching while you masturbate to internet porn.

But the implications go far beyond an individual’s personal porn predilections, or even rights to privacy.

Just because you might think you have nothing to hide, doesn’t mean the same applies to everyone else. There are people out there with legitimate reasons not to want their metadata tracked by the government. Like, for example, a government employee who stumbles across proof of government corruption as part of their job. Data retention makes it less likely a person in this situation would come forward, knowing their phone records and internet usage are constantly being monitored by the government, and that there are likely to be unpleasant personal consequences.

A strong democracy sometimes needs whisteblowers to uncover corruption or other inappropriate behavior by those in positions of power. Sometimes it is in the best interests of the whistleblower to remain anonymous, and for journalists to protect their sources. This becomes much more difficult, if not impossible, when mass government surveillance is in place. Which is probably the whole point. The government is far more interested is using data retention as a deterrent to whistleblowers (not to mention illegal downloaders) than terrorists.

Then there is plain old government stupidity. Believing you have nothing to hide if you haven’t done anything wrong is somewhat predicated in the false believe that governments — and all their employees and associated actors — are always sensible and rational and act intelligently. And most importantly, always make sensible laws.

This is hardly ever the case.

There are lots of laws, for starter. More than any one person can keep in their head, which is why we have specialist lawyers. How can you be sure you haven’t broken some law, if you’re not aware of every single law?

Everyone is guilty of something, no matter how trivial — or stupid.

And there are lots of stupid laws. For example, in Victoria: it’s illegal to be near or inside a house that is used regularly by thieves; only licensed electricians are allowed to change a light bulb; you can’t fly kites or play games in public which annoy people; you’re breaking the law if you have an article of disguise without a lawful excuse; you’re not allowed to sing an obscene song, tune or ballad within earshot of someone.

Who hasn’t been guilty of some or all of those things at some point?

Metadata retention is another stupid law, and gives more power to other laws, stupid and otherwise. Having a database of metadata gives authorities the possibly irresistible temptation to abuse their power in all kinds of creative ways. If a government agency decides to go after an individual — either vindictively, as in the case of a whistleblower, or simply because they mistakenly suspect an individual is guilty but don’t have proof — stupid laws and legal technicalities provide other means to hound that innocent person.

Having a database of metadata on everyone means the government will be able to arbitrarily find something to pin on just about anyone. If they desire to punish someone, the government will be able to trawl the metadata database in search of a crime or broken law that fits.

Just look at what happened to Freya Newman, the whistleblower who provided evidence that Prime Minister Tony Abbott’s daughter received a secret scholarship at a private design school. Abbott was opposition leader at the time, and his daughter, Frances Abbott, was his dependant. He did not disclose this scholarship, worth about $60000, and still hasn’t, despite disclosing far more trivial items. It appears the scholarship wasn’t advertised, hasn’t been offered since, and Frances Abbott received the scholarship after only a single meeting.

Regardless of the legality or propriety of it all, at the very least, the whole thing smells. The scholarship should have been public knowledge before the election. The Prime Minister should have disclosed it. The fact he didn’t makes it look even more suspicious. Especially since the private school in questions stood to benefit financially from the Liberal government’s education policies.

Despite this, Freya Newman was charged and found guilty. This is a disgraceful way to treat a whistleblower who disclosed information clearly in the public interest. Thankfully, a small measure of sanity prevailed, and today a Magistrate sentenced Freya Newman to a two-year good behaviour bond, with no conviction recorded.

What Freya Newman did might be technically illegal but it wasn’t wrong. She broke a stupid law, and for good reason. She did something even many legal experts didn’t realise was illegal to reveal private information the public had a right to know.

The legal system was then used to vindictively punish Freya Newman for revealing information that was inconvenient to the Prime Minister.

It’s an example of existing stupid laws being used for dubious intention by the state.

Mandatory data retention will give the government phenomenally more power and scope to do this. And not just the current government, but any future government.

Imagine the kind of thing a vindictive government could do with years of phone and internet records of an individual who does something that displeases them? Even if the individual’s actions are justified?

Mandatory data retention is a serious concern to whistleblowers, activists and journalists. But in an online, social-media-connected world, where it’s becoming almost impossible to use the internet without breaking some law, it should be a concern to all internet users.

With the government demonstrating they can and will use the legal system to punish the innocent, and the prospect of government agencies watching your every online move, do you really reckon you have nothing to hide?

They’re coming for us and our metadata

2014-11-22T00:00:00+11:00

They’re coming for us.

Them — the politicians, bureaucrats, lawyers.

They coming for us — programmer, developers, designers, scientists, engineers. Us geeks.

They’re coming for us, for our freedoms, and for our privacy, and they’re calling this “mandatory data retention”. Or metadata. Or “national security”. But regardless of what they call it, it is in fact just an outright attack on our democratic rights and freedoms.

They’re coming for us. To maintain their power and influence. And stifle ours.

It’s a war. Which They are winning. And we are losing quite badly.

Especially in Australia.

The Abbott government has managed to successfully screw us out of a decent Fibre-based broadband network (the only thing the previous government got right). They screwed up the NBN, so now they’ve got their sights set on screwing up the technology industry in other ways.

They’re doing this under the pretext of reforming “national security” legislation, but calling it that gives it credibility is doesn’t deserve, because it will do little to combat the risk of terrorism. (A real risk, but realistically small in Australia, certainly not high enough for draconian legislation that takes away democratic freedoms.) It’s far more accurate to call it “anti-democracy” legislation.

The first tranche of legislation was an attack on freedom of the press. It threatens anyone who says or writes anything about an ASIO special operation — journalist, blogger or social media user — with up to 10 years in jail. And anything can be deemed an ASIO special operation. It also gives ASIO the power to spy on the entire internet with a single warrant, and to plant spyware on people’s computers.

The second tranche further restricts freedom of speech, and sets a disturbing precedent for essentially treating people as guilty unless they can prove they are innocent.

The third batch of regulation has the technical community in its sights. It is a full on attack on the internet, and it users. On us.

The government wants to force ISPs to retain customer “metadata” — to store phone and internet usage data on all their customers. They want ISPs to spy on us on behalf of the government. Spy on our phone calls, and on our internet usage.

Apart from being creepy, there are many reasons this is a bad idea.

It will be expensive, and the cost will inevitably be passed on to us. This will essentially be an “internet tax”, in that it will make internet access more expensive in Australia. And Australia already has expensive yet crappy internet.

It is a gross invasion of privacy. And it isn’t even clear who will end up having access to this metadata. It seems likely that it will be used as evidence to prosecute online piracy, as well as being used in all kinds of civil lawsuits.

But most disturbingly, it will have the opposite effect to that intended. It will actually make us all less safe. This stored metadata will be an irresistible honeypot for hackers the world over. It means hackers will inevitably get access to all sorts of sensitive metadata, of both individuals and government employees. Best case scenario this hacked metadata is used to embarrass the politicians who voted for it. Worst case is it’s used to blackmail people in positions of power.

Mandatory data retention will do little to make us safer, but it will make the internet more expensive, slower, less secure, and less private. It’s win for government, but a loss for everyone else.

We are basically losing a war against politicians trying to take away our freedoms, making things more difficult for us, and generally meddling in an industry they don’t understand.

They’re winning.

What’s worse is not just that they’re winning, but that they are clearly idiots with delusions of competence. The worst thing is that we are losing to a bunch of out of touch and ill-informed idiots.

This has been obvious in the past few months.

Attorney-General George Brandis threw the first turd in the metadata shitfight when he did an interview on Sky News to announce the government’s intention to introduce mandatory data retention.

During this interview, Brandis was unable to explain what metadata actually is. He clearly had no idea. This is the guy who spend thousands of dollars of taxpayer money to build fancy bookcases for his office. It seems none of those of those books had a serviceable definition of metadata.

As funny as that trainwreck of an interview is (and it is funny), it’s also depressing because of what it implies. There is no positive way of looking at it. It’s disturbing no matter which angle you take.

The most charitable interpretation is that George Brandis is simply an idiot. But if the George Brandis is a simpleton theory is too simplistic, then it implies that he didn’t care enough about the issues to do even some basic research into the topic before going on TV to talk about it. There he was, talking about introducing draconian legislation likely to severely impact millions of people, not to mention introduce huge costs to many businesses, and not across any of the issues. Not even on a really basic level. (If he’d at least said something like “source IP address” or “DHCP logs”, even if somewhat out of context so it was obvious he didn’t really understand the specifics, it would have at least shown he had made some effort to understand the technical details. It would have at least shown he was professional enough to do some fucking research. But all he could manage was to incoherently mumble about “web addresses”, which means nothing, and shows how little he cares about doing his job properly.)

If the Attorney General can’t be bothered doing even some basic research on technical terminology, it’s probably an indication of how little everyone else in the government really cares about the subtleties of this anti-democracy legislation they’re planning. And of how little this government care about technology industries in general.

The government would like nothing better than to ram all this legislation through parliament with little debate, and even less thought about the consequences. They are meddling in an area they don’t understand. And they clearly don’t care.

Obviously there aren’t enough votes in this for them to care.

Which I guess is why it’s important for us, those of us who do care about such thing, to spread the word about how terrible the decisions our political masters are making.

I would have thought Brandis’s disastrous metadata interview would have had some consequences, possibly derailed the government’s plans a bit. But it didn’t seem to, because shortly after followed a unbelievable sequence of stupidity from senior government ministers:

Treasurer Joe Hockey revealed he thinks poor people don’t drive cars.

Senator Eric Abetz revealed he thinks abortion causes breast cancer. (Based on research from the 1950’s, apparently, which makes one wonder what decade he thinks we’re in now?)

The metadata stupidity was lost in the mix. It’s as if we have a government governing through abject stupidity and constant fuck ups. As First Dog on the Moon so aptly put it, it’s Operation Ongoing Enormous Clusterfuck Is The New Normal.

The weird thing is, it seems to be working. They’re winning. The government managed to ram through the first two tranches of anti-democracy legislation quickly and with little debate. They tried to do the same with the metadata legislation, but thankfully failed, which is one small, tiny win for us. But it looks likely to get passed early next year.

The idiots are going to win.

Unless we can make the mainstream see what a terrible idea it is. Unless we can convince enough voters (or enough voters in marginal electorates) that there will be a tangible downside of having the government constantly spying on them, and turning ISPs into tools of state surveillance.

Unless we can make the government care by making sure they lose votes because of this, and possibly make sure their campaign donors lose out in some way. It’s going to take a lot more than just tweets and bitching about it on social media, or making clever memes.

What are the chances we can pull that off?

We’re screwed. The idiots are going to win again.

To An Australia That Cares

2014-08-31T00:00:00+10:00

We interrupt our usual technical trivialities to bring you an important open letter from Rich Corporate Douche, CEO and Chairman of Corporate Douche Multiplex Cinemas Ltd, who wants you to feel really sorry for him and his important rich person problems.

From the desk of Rich Corporate Douche:

Dear Pirating Plebs,

A young man born in the Sydney suburb of Douchetown, I have become one of the world’s finest crony capitalists. Along with my fellow corporate cronies, and with the assistance of compliant governments, we have conspired to eliminate major competitors in the Australian movie business and build a cosy entertainment oligopoly. Our extraordinary efforts at subverting free market forces have earned us millions.

With little real consumer choice in the market, this has allowed us to build a profitable business model based around treating our customers with contempt. We can charge them whatever we want for movie ticket prices and food. We can make them wait months, if not years, to see movies already released overseas. Even if a movie happens to be made in Australia, overseas audiences will see it long before it’s released locally. They are all shining examples of how easy it is to exploit the Australian creative industry.

We love technology when it increases our profit margins, because those profits can be used to fund expensive personal drug habits. For example, technology advancements allowed us to automate many aspects of our cinema business and thereby reduce staff. Less money spent on staff means more profits, and more profits means more drugs for us, so it’s a win-win situation. Unless you’re a customer watching a movie and a technical glitch causes the sound to drop out at a crucial point in the movie, and there is no staff member to complain to, yet alone fix the problem. But we don’t care about any of that as long as we get our drugs. It’s this focus on profits over the customer experience which I’m sure will make you proud of the Australian industry.

All this wonderful Australian creativity and excellence in profitably screwing our customers is in real jeopardy of being lost if the problems the internet poses to our business isn’t solved. We hate technology when it creates competition for our lazy and outdated business models, because it’s really hard to innovate and compete with expensive personal drug habits to maintain. It’s OK when we used technology to cut costs and increase profits, but it’s evil and illegal when consumers use technology to bypass our cynical and inept business practices. After decades of treating customers with contempt, how dare those customers bypass our oligopolies — how dare they consume media online instead of going to our cinemas and buying ridiculously overpriced popcorn!

As rich people who have made lots of money and monopolized an industry for a really long time now, we feel entitled to continue to make money and monopolize the industry in perpetuity. We are entitled to our profits. But this is the good kind of entitlement — rich people entitlement. The Government have rightly moved to end the age of entitlement, but that only applies to poor people. We’re rich white men with political connections, so it’s different. We are entitled to our profits, and neither the free market nor technology changes will stand in our way when we can use our money and power to make politicians skew the playing field in our favour.

Adjusting business models for changing technological landscapes and adapting to profound cultural shifts in the way people consume media content is just not the way things are done in Australian business. Especially when it’s so much easier to bribe politicians to regulate away competitive threats to our monopolistic market share.

Online content consumption, even the legal kind, is a threat to our increasingly outdated business model, and therefore by extension, a threat to our shareholders, trophy wives, mistresses, kids, and personal drug habits. Especially personal drug habits. Because our position on piracy issues is clearly the result of overindulging in illicit substances.

Being rich has many advantages, including being able to afford some really great drugs. While under the influence of those drugs we came up with a potential solution to the problems online piracy poses to our business. After lots of smoking funny stuff, the solution became so obvious we were surprised no one had seriously proposed it before:

Let’s make the internet illegal.

The Government, through Attorney General George Brandis and Minister for Communications Malcolm Turnbull, have moved positively to address this crucial issue. But they haven’t done anywhere near enough. We still don’t feel we’ve got value for money with regards to our many corporate donations to the Liberal Party. The Liberal Party still owes us, so we want The Liberal Party to take immediate steps to make the internet illegal.

Those calling for a nuanced, common sense approach to online piracy are clearly crazies whose hidden agenda is theft of movies. Those who insist on trying to explain matters regarding science or technology to senior ministers in this government are even crazier. And anyone who thinks we’re going to get any decent technology related policy out of this government is downright delusional.

Given than both Prime Minister Tony Abbott and Attorney General George Brandis have demonstrated they know nothing about the internet, it will be much simpler to convince them to ban the internet entirely, rather than trying to explain it to them.

We believe that Australia cares and it is a choice between a tiny group of out of touch rich white men unable to adapt to the digital age, and the rights and freedoms of millions of Australian internet users. I’m sure the Australian people understand the needs of rich people should come first. After all, the jobs of 900,000 people and economic value to our economy of $90 billion annually depend on us. And that’s just our mistresses and drug dealers. Tragically, it could also mean the next generation growing up without knowing what it’s like to pay a week’s wages for a small popcorn and coke.

This won’t happen. We care. Mostly about our profits. And expensive drug habits. And also shareholders. But mostly the drugs.

Which is why, after making the internet illegal, we want the government to make computer games illegal too. Because computer games compete with movies for people’s attention and entertainment dollar. And we really hate competition. We also want to outlaw YouTube cat videos too, while we’re at it. Can’t have people staying at home and watching cat videos on the internet for free instead of going to our cinemas. Internet cat videos are just as bad as piracy, they are stealing our potential income.

Also, in addition to making unemployed people apply for 40 jobs a month, we think the government should force every unemployed person to go see 40 movies a month at our cinemas too. That extra profit for us would be good for the economy, and great for our drug habits. (Exactly how unemployed people on the dole who have to apply for 40 jobs a month are going to find the time or money to go see 40 movies a month is an implementation detail for a lower pay grade, and not something for us rich elite to worry about.)

See? We’re just full of great ideas in the movie distribution business!

Like our exciting new joint venture with a rather mysterious but strangely compelling group calling themselves SATAN, or something like that. Their representatives are very keen to be working with us to capture the immortal soul of every patron to enter our cinemas. The details and business case are still being worked out, but we’re told we can buy lots of really great drugs with immortal souls, so needless to say we are very excited.

Sincerely,

Rich Corporate Douche
Chairman and Chief Executive Officer
Corporate Douche Multiplex Cinemas Limited

Piracy Propaganda

2014-08-26T00:00:00+10:00

Village Roadshow ran ads in newspapers over the weekend consisting of an open letter from its CEOs expressing much butthurt over piracy and “copyright theft”.

I really wish the language surrounding allegedly “illegal” downloading would change. It doesn’t really make much sense to talk about digital copies being “theft”. And from a legal perspective, there is actually no such thing as “copyright theft” — only copyright infringement.

But then, hyperbolic rhetoric is one of the tools the movie industry like to use against the competitive threat the online world represents.

Of course, the irony here is that regardless of terminology or language used, none of these nefarious “pirates” doing the dastardly digital deeds of “stealing” by making copies are likely to be reading a weekend newspaper.

Not to mention the hypocrisy of Village calling copyright infringement “theft” when their movie ticket and popcorn prices amount to highway robbery.

But, obviously, the purpose of the ad was less about preventing illegal downloading, and more an attempt to reframe the debate in a way that benefits them, while doubling as a promotion for an upcoming movie release.

The ad is all the more pathetic for being such obvious propaganda from an industry struggling to find relevance in the digital age. Village Roadshow is doing what crony capitalists usually do when faced with a problem of consumer choice or the free market reducing their profit margins — running to the government to help them out by further restricting consumer choice and market competition.

It’s regulation over innovation. The Australian way, when it comes to business.

So we have an emotive plea for the government to restrict the rights of internet users, under the pretext of saving movie companies from the scourge of people infringing their intellectual property.

Which, if you think about it, is a weird thing to complain about, and even weirder to spend thousands of dollars for the privilege of complaining about it in newspaper advertising.

They’re essentially complaining about the price signals the market is providing them. Illegal downloads are an indication that while some people might want their product, they’re not willing to pay the prices being asked. Or they want the product via a more convenient distribution model — digital download, or online streaming, perhaps viewed on a mobile device.

You would think if the movie industry had any idea about the basics of running a business, they’d spend more time working out how to best monetize these new consumer trends towards online content consumption, and less time lobbying governments to further skew they playing field in their favour. Or running newspaper ads claiming the sky is falling.

In a way, the movie companies are also complaining that many people are keen to consume the products they produce. A problem many industries would dearly like to have. It’s a sign the movie industry is doing something right. Or at least a sign their marketing departments are doing something right — doing an excellent job of convincing people they should consume their formulaic, cliched crap.

All of which suggest the real problem faced by the movie industry is not “piracy”, but changing consumer habits and preferences, driven partly by changes in technology. This would suggest the solution lies more in working out how to effectively monetize the trend to online consumption, rather than restricting the online rights and digital freedoms of the masses.

But that would be entirely too sensible for an industry too lazy to innovate. An an industry that has spent so much money on political donations, they’re determined to get their money’s worth.

Of course, those complaining the loudest about illegal downloads are movie distribution companies. They feel threatened not just by the very small percentage of their movies that are pirated online, but by the very internet itself. As more and more content moves to an online distribution model, their business model becomes increasingly obsolete.

It’s difficult to feel sorry for them though. They’ve had things pretty good for a long time, with their government sanctioned oligopolies. Now that the internet is undermining their monopolistic ways, they go running back to the government to protect their artificially inflated profit margins. They want the government to put as many artificial restrictions on the internet as possible, just so they can artificially prolong their profit margins in the face of changing technology and consumer behavior.

After decades of treating their customers with contempt (through high prices, poor customer service, releasing movies much later in Australia than other countries, etc) Village Roadshow are trying to portray themselves as the victim of the illegally downloading hordes, now that technology allows consumers the choice of bypassing their crappy business practices.

It’s weird and completely disingenuous for a such a soulless corporate entity as Village Roadshow to try and elicit sympathy out of this. After all, it’s not as if they’ve made much of a contribution to the movie going experience. Theirs is the mainstream multiplex experience that focuses on quantity rather than quality. Along with price gouging.

Compare and contrast this to a place like the The Astor Theatre, which provides a more “authentic” movie going experience, and one that many people actually care about. The two exist on opposite ends of a spectrum in many ways, and there’s a kind of weird irony that this ad ran on the same weekend that The Astor announced it will close in early 2015.

There were a lot of genuinely disappointed people after The Astor announced it would be closing. The emotion didn’t have to be forced or manipulated through full page newspaper ads. Can’t imagine anyone would really care if any individual Village cinema shut down. Or even the whole company.

I suspect on some level, Village Roadshow might be aware of this.

The emotional language of the ad also seems to reflect either a delusion about their importance to the local movie industry, or a weird insecurity about their place in it.

If Village Roadshow were to disappear, maybe the Australian movie industry would too. Or at least part of it. But I doubt it. If there is any money to be made, another company would step in to make it. Or to create a business model that works in a new, online era. It’s quite arrogant of Village Roadshow to think just because they can’t make money making Australian movies, then no one else can.

The fact that independent cinemas like The Astor have existed so long — especially given the many market forces working against them, not to mention competition from the likes of Village — suggests there does exist an audience passionate about films and the cinema experience.

Australia does care.

It’s just that Australia doesn’t care about the corporate tools at Village Roadshow.

How to install a Python development environment on Windows

2014-06-19T00:00:00+10:00

This is a brief outline of the process I used recently for setting up a Python development environment on a new Windows 8.1 machine. (Based mostly on information in these posts)

Installing Python itself on Windows — just to run Python scripts — is fairly straightforward: pick a Python implementation (CPython or one of the alternatives) download and run the installer, and you should be good to go.

But to do any kind of non-trivial development, you’re probably going to want a few more tools available — at the very least things like pip and virtualenv.

Since there is a really nice PowerShell implementation of virtualenvwrapper, I like to set things up so I can work from a PowerShell command line.

Here’s what I installed, in the order I installed it, to get a reasonably useable Python development environment on Windows.

Python

First step is to download and install Python.

I went with what I suppose is the conservative approach: 32-bit CPython 2.7. (Yeah, I know…)

After installing Python, make sure running scripts is enabled in PowerShell:

Set-ExecutionPolicy RemoteSigned

If the Python installer didn’t add the location of Python to the system path, you will need to do that yourself. I did that by creating file called profile.ps1 in ~/Documents/WindowsPowerShell and added the Python install path to $env:Path. My profile.ps1 looks like this:

$env:Path += ";C:\Python27;C:\Python27\Scripts"
Import-Module virtualenvwrapper

Modify the path as necessary for your system.

The second line, importing virtualenvwrapper, is explained below.

Setuptools

Next install Setuptools, a tool for building and distributing Python packages. It’s required by pip (see below). In the past we might have installed Distribute instead, but apparently Distribute has been merged into Setuptools, so now Setuptools is the way to go.

Download ez_setup.py and run it in an administrator PowerShell console.

pip

pip is a tool for installing and managing Python packages. It automates and simplifies the process of downloading and installing packages from online repositories (usually PyPI).

To install pip, download get-pip.py.

Then run it in an (administrator) PowerShell console:

python .\get-pip.py

virtualenv and virtualenvwrapper

Different Python project are likely to require different collections of packages. It’s common practice to avoid installing packages into the system Python installation, and instead use virtualenv to create isolated Python environments. virtualenvwrapper simplifies the use of virtualenv for common tasks, and there is a PowerShell port that works really well on Windows.

With setuptools and pip installed, we can now install virtualenv and virtualenvwrapper by using the following commands at a PowerShell prompt:

pip install virtualenv
pip install virtualenvwrapper-powershell
mkdir ~/.virtualenvs

To check if everything installed correctly:

Import-Module virtualenvwrapper
Get-Command *virtualenv*

This should produce a list of available virtualenv commands.

You should now be able to use virtualenvwrapper in pretty much the same way as on Linux (but there’s also PowerShell style cmdlets as alternatives).

To avoid having to type Import-Module virtualenvwrapper every time you open a new PowerShell window, it’s a good idea to add it to your profile, as shown in the earlier example.

Qt and PySide

My reasons for even bothering to install a Python development environment on a Windows machine usually have something to do with Qt based development. I use the PySide bindings, which on Windows can be installed in a virtualenv like so:

New-VirtualEnvironment pyside
easy_install -U PySide

The first line creates a virtualenv called “pyside”, and the second line installs the PySide bindings.

(The easy_install command didn’t work first time I tried it, not sure why. But it ran fine when I tried it again, after a reboot.)

How to turn a Windows box into a development machine

2014-06-12T00:00:00+10:00

One of the problems with a new Windows machine (or a clean install of an old one), apart from the fact it is a Windows machine, is the time that will inevitably be spent looking at progress bars while the many, many pieces of software a Windows PC needs to be usable as a development machine (or anything else more productive than web browsing device) are downloaded and installed.

The following are the things I installed recently on a new Windows 8.1 machine to suitably geekify it for my coding purposes, in approximate order of geekiness — from least technical (things all Windows machine will probably need to have installed, like browsers, document viewers, etc) to the most technical (IDEs and development tools).

Given a choice, I generally prefer to do development work on a Linux box. But sometimes having a Windows machine handy is a necessary evil.

This was actually my first experience with the much maligned Windows 8 UI. But I will reserve judgment until I’ve have a chance to use it for a while. My initial reaction so far is pretty much what I expected — Metro on a desktop is somewhat clunky and horrible, but nowhere near as bad as it’s been made out to be.

Browsers

Obvious first step is to avoid having anything to do with Internet Explorer.

So goodbye IE, hello Chrome and Firefox.

Document Viewers

Kindle for PC — for reading eBooks.

Adobe Reader for viewing PDF files — be sure to deselect any options for McAfee crapware.

Dropbox

You’re probably gonna want the desktop client from the Dropbox website, not the Metro app.

Backup

At this point I installed backup software and did an image of the hard drive, so that I’d have a clean and minimal image I could revert to if necessary.

Emacs

I’m an Emacs kinda guy, so even on Windows some kind of Emacs is compulsory.

After trying other variations, I inevitably seem to go back to the standard GNU build.

This time, I looked around for alternatives, and came across ErgoEmacs. I hadn’t heard of this variant before, but it wasn’t the “ergonomics” aspect that interested me — it was the fact it comes packaged with Windows ports of many common Unix commands, along with other configuration niceties. So I thought I’d give it a go.

Notepad++

Despite having Emacs on my Windows machines, I also like to have a more typical Windows text editor available, for those times I might want to just view files.

I find Notepad++ fills this need nicely.

Office

Despite being primarily a development machine, having an office suite installed can come in surprisingly handy, even if it’s just to view/manipulate CSV files in a spreadsheet.

Libre Office will usually do for those not wanting to go down the Microsoft Office route.

Python

A machine without Python is just not a development machine, as far as I’m concerned. But I will save the gory details of my Windows Python setup for it’s own post.

Java

Even if you have no intention of actually developing in Java at some point you’re likely to need at least the runtime, if not the whole SDK.

Be sure to opt out of any crapware options included in the installer.

Development Tools

So with the main preliminaries out of the way, it should now be safe to install actual development tools.

To each their own, but my Windows development needs at the moment revolve around a combination of Android and Qt.

For Android development there’s the choice of Android Studio or Eclipse with ADT.

These appears to work fine with Java SE Development Kit 7, even though Android is (currently) limited to the Java 6 API.

For any desktop UI work Qt is currently my preferred framework (usually via Python bindings). When I tried to install Qt Creator using the online installer., it got stuck while downloading files the first time round. But I ran it again and it worked. Go figure.

Linux!

Finally, because all too often on Windows I want to do something that is just much easier/faster/better done on Linux, I installed VirtualBox, then created a Linux Mint virtual machine.

How to make a Samsung Galaxy S3 play nice with Eclipse on Windows

2014-05-01T00:00:00+10:00

Using Eclipse on a Windows development machine, I’d managed to get USB debugging working on various Android devices with various versions of Android without too much drama. But I ran into some difficulties when I tried to debug an app on a Samsung Galaxy S3 (GT-I9300T with Android 4.3).

I think I’m beginning to detect a pattern emerging — the more recent the Android version, the more (unnecessary?) development hoops to jump through. (Because Security! Apparently.)

So here’s the full process I ended up going through to make everything work. The following worked for me on both a Windows 7 machine, and a Windows 8.1 laptop. (But as always, YMMV.) I suspect the process would be the same for the Galaxy S4 and S5, as well as other recent Android 4 devices, with the only difference being the USB driver required.

First thing, as usual, is to enable USB debugging on the device:

Open Settings, go to the More tab and select Developer options. If you don’t have any Developer Options then:
- Scroll down and Select About device.
- Scroll down to reveal Build number.
- Tap Build number seven times.
- Navigate back to the previous screen, and you should now find a selection for Developer options has magically appeared. Select it.
Enable the USB Debugging option.

When developing on Windows, an OEM USB driver is required. I installed the Windows USB driver from the Samsung website.

You may need to reboot at this point, especially if it’s a Windows 7 machine you’re using.

This next step is surprisingly significant. Before connecting the Galaxy S3 to the PC via USB, turn on the device and unlock the screen. Do whatever you need to do to prevent the phone from turning off/sleeping/locking until you’ve got Eclipse to recognise it.

With the phone on and the screen unlocked, connect it to the development machine with a USB cable.

Windows will then hopefully detect the device and do some driver installation. Wait for it to finish. But watch the screen on the phone.

At some point you should see a dialog pop up asking if you want to “Allow USB debugging” from this computer.

Select OK.

Eclipse (and adb) should now be able to connect to the Galaxy S3. If you look in the Devices tab in the DDMS perspective, you should see the device listed as “Online”:

If so, you should now be able to use Eclipse to run apps on the device in the usual way (Run as Android Application).

If not (if the phone comes up as “unauthorized” or “offline”), try switching the USB connection option on the phone from MTP mode to PTP, then back again. This should make Windows reload the drivers, and then the “Allow USB debugging” should appear.

It’s possibly worth pointing out at this point, if you find nothing works, try a different USB cable. In particular, keep the cable that came with the device handy (and labeled, if necessary!). I once wasted a lot of time trying to get Eclipse to connect to my S3 using a USB cable that just happened to by lying on my desk at the time, all to no avail. It would charge, but not do anything else. Turns out that one particular cable, for some reason, just doesn’t work. Switching to a different cable did the trick.

The Liberal Party’s broadband policy explained

2014-04-14T00:00:00+10:00

We interrupt our usual random technical trivialities to bring you an important announcement from the Minister for Mis-communication, Malcolm something-or-other, we’re not sure, we’d look it up, but our ADSL has dropped out again, and as usual Telstra refuses to fix it. Anyway, apparently this is what you need to know about the current state of broadband in Australia, and the Liberal Party’s grand technological vision for the country…

From the desk of The Minister for Mis-communication:

Dear nerds, journalists and welfare bludgers,

It appears once again my intentions are being misunderstood.

Once again allegations of hypocrisy and lying are being unfairly leveled against me, just because in opposition I consistently criticized the Labor government for proceeding with a Fibre To The Premises National Broadband Network without a cost-benefit analysis, and now in government I’m doing the same thing, and proceeding with my alternative Fibre To Nowhere Hodgepodge Broadband Network without a cost-benefit analysis.

I think there may be some confusion out there on where the Liberal Party of Australia — a division of News Corp (NASDAQ: NWS) — actually stands on broadband and technology issues. So let me try and clear things up for all you computer nerds and clueless tech journalists.

The Coalition’s Plan for better broadband is basically this: go work down the mines.

Forget about the internet, and just go work down the mines.

All you geeks and porn addicts and journalists — all of you put away your computers and phones and tablets, and go get a real job down the mines.

Because the only thing Australia is good for is mining. Just mining. That’s all our current and future prosperity should be linked to. We as a country should just focus on mining.

Because if there’s one thing the Global Financial Crisis taught us, is that there is no need to diversify or hedge risk if you’re onto a sure thing, and mining is as sure as houses and mortgage backed securities. And anyway, if something does go wrong, we now know any economic problem can be solved with clever financial engineering and endless money printing.

Because what could possibly go wrong anyway? After all, we are in a commodity super cycle that will never end. Australia can dig stuff out of the ground and sell it to the highest bidder forever.

Because we have so much stuff in the ground to dig up that we will never run out, and we will always have willing buyers.

So all of you geeks and bloggers and journalists — stop whining and go work down the mines

Australia is a mining country. We have no need for a modern broadband network, or a technology industry, or even technology. Australia needs fast and reliable broadband infrastructure about as much as it needs a car industry. Or an environment.

So the official position of The Liberal Party on broadband is that you should all go work down the mines.

Forget about the internet. Australia is a technological backwater and that’s the way it should be.

All we’re good for is digging stuff out of the ground and selling it to China, so that’s all we’re going to do.

That’s why we’re so keen to repeal the mining tax and the carbon tax — because all we care about is helping mining companies make as much money as possible. We don’t don’t care about other industries, especially not the technology industry. As if there’s money to be made in technology anyway. That’s why we have no interest in building a modern broadband network that would be a huge benefit to both consumers and businesses — because that would be a waste of money.

Those of you with ideas involving computers and the internet and technology startups — you’re all in the wrong country. We don’t need any of that, because we’ve got mining! Technology is what other countries do. We just do mining. Australia just digs stuff out of the ground. Then sells it to other countries.

Now I realise some of you may be slightly confused, because this might seem a bit different to what we in the Liberal Party promised before the election. We’re sorry you misunderstood us, but you can’t really blame us for your ignorance and lack of cognitive ability. After all, not everyone can be as smart as the Liberal Party.

Before the election, when the Liberal Party promised Australia would get a “fast” and “affordable” National Broadband Network “sooner”, what we really meant was: “fuck you, go work down the mines”.

I don’t know how the Australian public could have misinterpreted that.

If you’re confused, blame the liberal elite media. I know I do.

It was their fault, all those stupid tech journalists, falsely getting your hopes up by misrepresenting Liberal Party policies — mainly by reporting the Liberal Party actually had serious policies. It should be pretty obvious to everyone by now that we don’t, and never did. I’m sure those journalists won’t make the same mistake again.

Those critics accusing me of hypocrisy and dishonesty just can’t see the big picture. Our broadband policy is entirely consistent with our policy framework in other areas. For example, our policy for employment, which forces the unemployed to go work down the mines. And similarly the Liberal Party’s version of the National Disability Insurance Scheme (NDIS), which will force disabled people to go work down the mines.

Then there’s our education policy. I know what you’re probably thinking — that we’re going to force children to go work down the mines. Don’t be silly. That would be cruel and uncaring. The Australian Liberal Party is all about kindness and goodwill. That’s why as part of our new education policy, we will force children to work as Dickensian chimney sweeps. That will prepare them to go work down the mines when they grow up. It’s this kind of forward thinking that distinguishes the Liberal Party from all that profligate Labor scum. Everyone in the Liberal Party should get knighthoods for coming up with such progressive policies and being so in touch with the times.

So with everyone busy working down the mines, it should be obvious to all but the most obstinate nerds that no one will have time for the internet, thereby making a modern broadband network completely redundant.

Not to mention that we will introduce all manner of draconian legislation to severely restrict the digital rights and online freedoms of all Australians, so that using the internet essentially becomes illegal. Now there’s obviously no point investing billions of dollars to provide everyone in the country with a universally fast and reliable internet connection, if using the internet is against the law. Because it soon will be, now that the Liberal Party are in power.

So any Australians clamouring to download Game of Thrones — fuck off and go work down the mines. There shall be no popular TV show downloads for you while there’s important mining work to be done.

I know that some of you who voted for us might think it somewhat harsh and unfair that we’re forcing everyone to go work down the mines. But what you must understand is now that we have been elected by the people of Australia, we have a mandate to do whatever the hell we want, regardless of what we said before the election, and regardless of what you the people of Australia actually want us to do. That’s how democracy works.

Don’t bother reading the pointless ramblings of uninformed journalists and bloggers. Just accept the fact you elected us and now we will make you all go work down the mines.

The sooner all these stupid journalists in the liberal media get on board with my broadband plan — and go work down the mines instead of writing uninformed drivel — the better it will be for all concerned. After all, they seem quite worked up about how badly I’m screwing up broadband in this country. I honestly think they will be happier to forget about the whole broadband thing and just go work down the mines.

I hope that clears things up.

Now stop reading this shit and go work down the mines.

Vote Liberal bitches.

Sincerely,

Sir Malcolm
Duke of Wentworth
Minister for Mis-communication

Introducing VoidyBootstrap: A theme for Pelican

2014-03-30T00:00:00+11:00

So I kind of accidentally made a Pelican theme.

Didn’t mean to. Really I didn’t.

At least, I honestly had no intention of engaging in this kind of geekiness when I started this blog.

I just wanted to write a blog. A geek blog. I set out with every intention of ignoring the blogging technology and resisting any urges to tinker with themes (or anything else) and just write.

I failed.

I just couldn’t resist.

A little change here. Then a little change there. And pretty soon I had this monstrosity of a Bootstrap based Pelican theme on my hands.

Yes, another Bootstrap theme. Because what the world really needs is more websites with the now ubiquitous Bootstrap look. But in my defence I have tried to make this theme as easily customisable as possible. Which is the purpose of this blog post: to justify the existence of yet another Bootstrap blog theme, by demonstrating how (hopefully) easy it is to use the VoidyBootstrap theme to make a website with Pelican that doesn’t obviously look like a Bootstrap website.

I feel I can’t just unleash this (somewhat idiosyncratic) theme without at least attempting to justify its existence (and idiosyncrasies). So here goes.

When I started using Pelican for this blog, despite my limited design skills, I managed to hack together a custom theme that made use of Bootstrap 2 and Font Awesome 3. When I decided the time had come to upgrade the theme for Bootstrap 3 and Font Awesome 4, I took a step back first to evaluate how I wanted to proceed.

I used the break of backward compatibility that Bootstrap 3 presented as an opportunity to rethink my design and approach. (And I use the word “design” in this context very loosely.) Upgrading the theme for Bootstrap 3 necessitated somewhat of a redesign (or at least a bit of work), so I decided if I’m going to do this, I only want to do have to do it once.

If I were to use the combination of Pelican and Bootstrap to create other websites, I didn’t want to have to start from scratch.

Of course, there’s always the option of doing what seems be the standard practice for creating a custom website with Pelican: looking through the official themes repository, selecting a theme with a design that’s closest to what you want to achieve, then hacking away with wild abandon at the theme’s Jinja templates and CSS to implement the desired results.

Which is all well and good: Pelican is a technical tool for a technical audience who often have the skill and desire to do just that. But while there are a lot more themes available now than when I started playing with Pelican, few seem to be designed with customisation in mind.

I thought it might be worthwhile to create a very basic Bootstrap theme, with minimal additional styling, that would act as a reasonable starting point for a Bootstrap website with Pelican that could be easily customised — so that it doesn’t obviously look like a Bootstrap website!

The result is VoidyBootstrap. It’s intended more as a “meta-theme” than a stand alone theme (although it can certainly be used as a stand alone theme, and should work fine without any customising). It’s a basic blog theme based on Bootstrap — but one that (hopefully) facilitates customisation as painlessly as possible.

UPDATE, November 2014: This post is about version 1 of the VoidyBootstrap theme. It’s been reworked a bit since then, and many of the configuration settings changed. See the theme’s latest README file and changelog for details.

UPDATE, June 2015: While the concepts in the following sections are still mostly relevant, the details have changed. In version 2 of VoidyBootstrap, many of the configuration settings described below have been renamed. There is a summary of the major changes in this post.

VoidyBootstrap Quickstart

While the main goal of VoidyBootstrap is easy customisation, I also wanted to make things as easy as possible for those who just want a simple blog theme and don’t want to mess around with theme modifications. So for anyone happy with a fairly stock standard default Bootstrap look (at least initially), the VoidyBootstrap theme gives you that with a minimum of effort (hopefully).

The theme should work without any theme-specific options configured (I think). But as a bare minimum, you’d probably want to configure the following theme-specific settings in your pelicanconf.py:

SITENAME = 'mywebsitename'
SITESUBTITLE = 'Sub-title that goes underneath site name in jumbotron.'

# Extra stylesheets, for bootstrap overrides or additional styling.
STYLESHEETS = ("pygment.css", "voidybootstrap.css",)

# Use the default sharing button implementation.
CUSTOM_ARTICLE_SHARING = "sharing.html"
CUSTOM_ARTICLE_SCRIPTS = "sharing_scripts.html"

The following settings from the default Pelican theme are supported:

SITEURL
FEED_DOMAIN
FEED_ALL_ATOM
FEED_ALL_RSS
DISQUS_SITENAME
GOOGLE_ANALYTICS

Configure those that you require (usually in publishconf.py).

(See the theme’s README for a more detailed configuration example.)

And that’s pretty much it. With that you get a blog layout that looks something like this:

What’s provided by default is a responsive 2 column layout that stacks vertically on small screens. There is a navbar and Jumbotron at the top, and the sidebar contains category links and a tag cloud.

Customising Styles

You may notice the theme’s static/css directory is relatively bare. This is intentional. By default, all the CSS and Javascript files required by the theme (Bootstrap, Font Awesome, JQuery, etc) are downloaded from CDNs.

So by default the only styling provided by this theme is from a standard bootstrap.min.css downloaded via a CDN. And nothing else — no CSS files from the theme’s static/css directory are used, unless configured otherwise (which I’ll explain in a moment).

VoidyBootstrap provides two ways to customise the theme’s unashamedly default Bootstrap look (without having to resort to hacking at the supplied templates or CSS files). These methods can be used individually or in conjunction with each other.

The first is the STYLESHEETS setting. This is an array for specifying any additional stylesheet files. As mentioned above, by default the theme grabs what it need from CDNs and doesn’t reference any CSS files in static/css. The STYLESHEETS setting is provided to enable any additional (local) CSS files to be explicitly configured. It should be set to a list of CSS files present in the theme’s static/css directory.

All the CSS files in the STYLESHEETS array will be referenced by the theme’s base.html template after the main Bootstrap CSS file, so this provides a way to override the default Bootstrap styles. Therefore a simple way to provide your own styling on top of this theme is to add your own CSS files to the static/css directory, and add their filenames to the STYLESHEETS array in pelicanconf.py.

The theme’s static/css directory contains a sample CSS file (voidybootstrap.css), which can be used as a starting point. It provides a minimal set of styling (mostly just spacing and whitespace changes).

Of course, because STYLESHEETS is an array, any number of CSS files can be included, and it can be used to bring in any additional custom styling (not just Bootstrap overrides).

The second method of customising the theme’s default look is the MAIN_LOCAL_STYLESHEET setting. This configuration option allows the standard Bootstrap CSS file to be replaced with a customised one. It’s provided to make it easy to do things like integrate a customised Bootstrap file generated by the Bootstrap customizer, or one built using an advanced Less workflow.

Simply place the customised Bootstrap CSS file in the theme’s static/css directory, and in pelicanconf.py set MAIN_LOCAL_STYLESHEET to the filename.

(If the MAIN_LOCAL_STYLESHEET option is not set, the standard bootstrap.min.css will be used — from a CDN — as described above.)

With the combination of STYLESHEETS and MAIN_LOCAL_STYLESHEET settings, there’s the flexibility to accommodate different degrees of customisation and various Bootstrap workflows. For example, to incorporate CSS from one of the Bootswatch themes, it should be a relatively straightforward matter of downloading the CSS files, placing them in static/css and configuring the STYLESHEETS and MAIN_LOCAL_STYLESHEET settings in pelicanconf.py as appropriate.

Adding Custom Content

Conceptually, Pelican themes are quite simple (although as is often the case, the devil is in the details).

A Pelican theme usually has two directories: static and templates.

The static directory is where any CSS or image files required by the theme will go (along with any other static assets).

The templates directory will usually contain HTML files, which are actually Jinja2 templates.

Pelican requires certain templates that must be present. These in turn can include other templates. To help keep things organised, VoidyBootstrap has all templates included from the core templates in an includes subdirectory.

Like with any other Pelican theme, you are of course free to modify any of the theme’s templates to implement a different layout and add custom content. But if you’re happy to work with the layout the theme provides, and all you want to do is add a few sections of custom content here and there, then VoidyBootstrap provides a potentially easier/neater way to do this.

VoidyBootstrap takes advantage of the Jinja “include” statement with the “ignore missing” option. At strategic points in the theme’s layout, it attempts to include “optional” template fragments (i.e. templates which are used if they are there, but the theme works fine without them, and they are not supplied by default). Therefore it’s possible to add chunks of arbitrary custom content by simply creating these optional template files in the theme’s templates/includes directory.

This is all in an attempt to cater for some common customisation needs. Of course, no one theme can anticipate every design need. So for things like extensive layout changes, modifying the templates is the only option. But I wanted to provide a middle ground — I wanted VoidyBootstrap to provide ways to make simple customisations to content and appearance without having to delve too deep into Pelican’s theme template system.

The diagram below gives an overview of the the templates used to render an article page. The dashed boxes represent these optional templates.

For example, let’s say that at the top of the sidebar of your website you’d like to have an “About” section. You could of course just edit sidebar.html. But adding custom content to the sidebar would be a fairly common requirement when creating a website, so VoidyBootstrap provides some “hooks” to make this a bit easier (not just easier to implement, but also easier to keep your customisations separate from the theme’s templates, for those that might care about that).

Notice in the above diagram that sidebar.html provides two optional templates: sidebar_top.html and sidebar_bottom.html. To have your custom content appear at the top of the sidebar, create a template with the required content in the includes directory called sidebar_top.html. The contents of that template fragment will then appear at the top of the sidebar.

As another example, if you decide you’d like a navigation element at the top of article pages, you could place the HTML for this navigation element in a template called article_top.html. Want to add a copyright message to the footer? Just put it in a template called footer.html and it’ll be automatically included.

See the “Optional Templates” section in the README for a list of all the available optional templates.

Overriding Existing Content

In addition to being able to add new content, the VoidyBootstrap theme also allows some existing sections of content to be swapped out by a custom implementation. It’s an attempt to cater for the case where the theme satisfies most design needs…except for that one thing…

As an example, let’s once again consider what would be a fairly common requirement: customising the sidebar. The default sidebar provided by the theme displays social links (if configured), categories and a tag cloud. The previous section described a way of adding custom sidebar content in addition to what’s there by default. But what if you don’t want categories and a tag cloud in the sidebar? What if you decide that’s completely inappropriate for your website? What if instead you want to have a sidebar full of cat pictures? Well the VoidyBootstrap theme has got you covered. It won’t judge you for what you decide to put in your website’s sidebar, and in fact will gladly facilitate a completely customised sidebar. Simply create your glorious feline sidebar template and put it in the theme’s templates/includes directory (or subdirectory thereof). You can call it whatever you want (as long it doesn’t conflict with anything already there, or the name of an optional template). Let’s say you’ve called the new template cats_sidebar.html, and placed it in a subdirectory called custom. To enable your custom cat pic sidebar, in your pelicanconf.py file add the following line:

CUSTOM_SIDEBAR = "custom/cats_sidebar.html"

This will result in the theme’s implementation of the sidebar being completely replaced with whatever is in cats_sidebar.html. (Goodbye tag cloud, hello kitties!)

VoidyBootstrap provides a number of these “Custom Includes” that allow parts of the theme to be overridden by an alternative implementation. (See the “Custom Includes” section in the README for details.)

As another example, if you think the theme’s article headings are an abomination against all that is good and holy in web design, you’re able to supply your own. Not only that, if you want you can supply different article heading templates for index pages and article pages, thereby allowing for differentiation in layout and style. Simply create templates for custom article headings that please your aesthetic sensibilities, then set the CUSTOM_ARTICLE_HEADER and CUSTOM_ARTICLE_HEADER_INDEX settings appropriately in pelicanconf.py.

Conclusion

There’s more to the VoidyBootstrap theme than what’s been mentioned here. While being a deliberately minimal theme as far as design goes, it aims to be fairly feature complete in every other respect. It’s a modern Pelican blog theme, with hopefully all the features you’d expect in a modern blog theme, and even some surprisingly nice features you might not expect (like custom metadata tags, “featured images”, Open Graph, Twitter Summary Card metadata, and more!).

But it’s main reason for existing are the customisation features discussed in this post. It’s a Pelican theme based on Bootstrap that’s designed to be customised. To that extent, it tries to anticipate and facilitate a range of customisation needs. How appropriate the theme is for your needs will most likely depend on how you feel about usefulness of these customisation abilities. (Or how much you like the default Bootstrap look.)

The winding road to a workable writing workflow

2014-03-19T00:00:00+11:00

It’s hard to imagine now that early incarnations of this blog started as a Wordpress site.

Not that I have anything against Wordpress itself, as such. It’s just that I’ve developed a deep dislike for PHP (the underlying technology used to develop Wordpress). And I’m not the only one.

My brief experiences with PHP were enough to realise it’s pretty damn horrible. I feel I know enough about PHP to know I never want to go anywhere near PHP ever again, if I can help it. That thankfully short but deeply unsatisfying dalliance with PHP sent me on a search for a better match my web development desires, culminating in a much more rewarding tryst with Django, and occasional flirtations with Flask.

So it was perhaps strange that I initially considered using Wordpress for a personal geek blog. But there was some method to my madness: I actually hoped the underlying ugliness of PHP might, somewhat paradoxically, be a benefit in this case.

I’ve started blogs before, with all the best intentions of regularly writing something (anything!), but they all went no where, languishing for months without updates, then years, before eventually being scraped and taken down. But I think in those initial failed attempts at blogging I did learn a few things about myself, and the strange love-hate relationship I have with the writing process.

I realised one of the side effects of being prone to a sometimes counter-productive combination of procrastination, perfectionism and nerdy proclivities is that it’s all too easy to be distracted by things that are not actually writing. Like the blogging technology itself. Or the blog theme. Or some other technical detail. Which means a lot of technical tinkering happens, but very little writing gets done.

Not that I ever had any great ambitions as a blogger. When approaching blogging, I have set myself the ultra-low benchmark of just writing for the sake of writing. Who knows, maybe if I write enough, I might occasionally write something worthwhile. One day. Eventually. And if nothing else, there might (hopefully) be some purely personal benefit in taking the random thoughts floating around in the chaos of my mind, imposing some semblance of order and writing them down.

Regardless of whether anyone reads them.

The main thing is just to write.

Because a blog without some attempt at decent content is surely pointless. Unless you’re an internet marketer, in which case apparently content is irrelevant and it’s all about SEO tricks and link building.

But I still cling to some semblance old school ideals about aiming for at least half decent (original) content. So this time I was determined to start a technical blog and keep it going. Just my own little place on the interweb to call my own. A place to air out my half baked ideas. Which means actually writing regularly.

I actually hoped that using Wordpress would enable me to get more writing done simply as a virtue of my complete lack of technical interest in Wordpress as software, and complete disdain for PHP as a programming environment. I was somewhat naively hoping this would mean I’d be less tempted to tinker with the technology in any way, which in theory would mean I’d be less likely to get distracted from writing.

The idea was just to use Wordpress as a tool, a means to an end. All I had to do was set it up, configure it, select a theme I could live with without having to modify it in any way, and start writing.

Which is what I did. Although that selecting a theme thing took a really long time. The great thing about a mature blogging platform like Wordpress is that there are a lot of themes to choose from. The downside about a mature blogging platform like Wordpress is that there are a lot of themes to choose from. An option paralysis inducing amount. It’s hard enough to make a decision amongst so many different choices. It was even harder since I’d given myself the restriction that I was just going to use the theme as is — I was not going to waste time customising it in any way. Because customising a Wordpress theme might mean being exposed to the horrors of PHP, which is something I wanted to avoid at all cost

I should have known that was never going to happen. I set up Wordpress and finally decided on a theme. But I could not resist the urge to customise. As usual, it started by changing a thing a little thing here and there. Then before I knew it, I was knee deep in the guts of Wordpress, writing PHP code.

It was then that I decided Wordpress was not for me. If I was going to tinker, I wanted to tinker with technology I actually enjoyed tinkering with. So I scrapped the Wordpress idea and went looking for alternatives.

I found many viable alternatives in the nascent world of static site generators that were beginning to appear. Thankfully, by this time, dynamically generated, database backed blogs were no longer the only blogging game in town. By now, it seems many bloggers (especially geek bloggers) had realised that in some cases, like single author blogs not frequently updated and written by technically adept command line junkies, running a database and dynamically generating rarely changed pages for each request was somewhat redundant. Not to mention that for those of us comfortable using powerful text editors like Emacs (and even vi), typing blog entries into a tiny text box on a web page is not exactly an optimal workflow.

So there were lots of static site generators appearing. These make a lot of sense for technical blogs. Writing entries as text files in Markdown or reStructuredText format, and tracking them using a version control tool, is a familiar workflow for developers. And statically generating the blog simplifies and speeds things up immensely.

I explored what was available at the time, and the most mature at that point seemed to be Octopress. However, I’m not a Ruby guy. Yet. I like what I’ve seen of Ruby so far, and have every intention of exploring it further at some point, but for now I’m still having a lot of fun getting better acquainted with Python.

Octopress seemed to be exactly what I wanted though, with some reasonably good themes to choose from. And once again, the fact the blogging engine used technology I wasn’t very familiar with was potentially a benefit, in that I was less likely to distract myself from writing by tinkering with it. The downside of not being familiar with the underlying technology, however, is that the learning curve is much steeper, and any problems take much longer to solve. This in addition to the overhead of having to install and maintain a Ruby environment which I otherwise have no need for.

So I explored further, and found there were a few Python static site generators around, although none seemed quite as mature at Octopress. A Python static site generator would be perfect, since for starters, as a Python developer, I was already likely to have a Python development environment set up and good to go.

The three obvious choices at the time came down to Hyde, BlogOFile, and Pelican. (I realise there’s a lot more now.) I ended up choosing Pelican: I found it fairly straightforward to set up a blog, it looked like it was being actively maintained, and there was a reasonable selection of themes available.

But of course, there was no theme I was completely happy with.

Fortunately it wasn’t too hard to figure out how Pelican themes work, and hack together something more or less to my liking. What I ended up with was a hacky custom theme based around Bootstrap 2 and using Font Awesome 3 for icons. It might not have been too pretty, either under the hood or from a design point of view, but it looked more or less like I thought a blog should look, with all the common features I thought a modern blog (especially a technical blog) should have (sidebar, categories, tag cloud, decent navigation, etc). After much tinkering I got the theme to the point where I was happy enough to leave it alone and just write.

Then Bootstrap 3 was released.

Bootstrap 3 deliberately broke backward compatibility with Bootstrap 2.

This meant a substantial amount of tinkering would be required to upgrade my custom theme. One of the downsides of tinkering, and why I initially resisted the urge to customise things. Because this is what tends to happen once you start tinkering with technology. Things change, things break, things need upgrading. These things take time. Time possibly better spent writing, which is after all the point of a blog (unless you’re an internet marketer).

But I resisted the urge to tinker, and stuck with my custom Bootstrap 2 theme for the time being.

Then Font Awesome 4 was released. It also broke compatibility with the previous version.

So eventually I decided to set aside some time to work on my hacky custom theme, and update if for both Bootstrap 3 and Font Awesome 4. While I was at it, I figured this would be a good time to clean things up a bit, and try to impose some order on my previously hacky ways.

The result is VoidyBootstrap.

A deliberately generic blog theme for Pelican based on Bootstrap 3, but designed to make it easy (or at least easier) to create a customised (and less generic) look and feel.

It’s a theme that I created when I should have been writing. A theme born of procrastination.

I’m not sure if that’s a sign of my lack of discipline, or testament to my geek cred.

It’s kind of ironic that I started this blog with every intention of just focusing on the writing part of blogging, but ended up spending a lot of time messing around with the blogging engine and front-end frameworks, and ended up creating a possibly over complicated blog theme in the process.

I’m not sure if there’s a moral of the story to be found in there somewhere. Or a point to all this rambling. Maybe it’s just that hackers are gonna hack.

Maybe it was naive of me to think that I would be able to blog without getting side tracked by blogging technology and themes. Maybe I should just accept the fact that as a geek, if I’m going to be blogging, I’m not just going to be writing — I’m inevitably going to be tinkering with technical things as well. Because that’s just me.

Perhaps is just the nature of the geek beast.

And perhaps if nothing else, it shows that amongst PHP’s many failings, it’s also useless as a psychological ploy to avoid procrastination.

Upgrading to Pelican 3.3

2014-01-23T00:00:00+11:00

Maybe it’s just me, or maybe it comes with the territory of being a geek with a blog, but seems I often have to fight the urge to tinker with the blogging technology I’m using — in my case Pelican — in order to get any actual blogging done.

Initially, the tinkering took the form of endlessly tweaking the custom theme I’d hacked together based on Bootstrap 2. Eventually the theme got to the point where I was happy enough with it to actually leave it alone and get some writing done.

Then Bootstrap 3 came out, deliberately breaking backward compatibility with version 2, which meant upgrading my theme would require substantial tinkering. But I resisted, and stayed with the Bootstrap 2 theme for the time being. Then Pelican 3.3 came out, and once again I resisted upgrading, just in case I’d have to deal with time-consuming upgrade issues. Then Font Awesome 4 came out, also breaking compatibility with the previous version.

Eventually, I could resist the urge to tinker no longer, and decided to do the work to upgrade all three in one time-sucking swoop. Starting with the Pelican upgrade.

So it was with the usual slight sense of trepidation that I fired up my virtualenv and gave the command:

pip install --upgrade pelican

Turned out my trepidation was somewhat justified. Things broke.

Pelican 3.3 devserver issues

After doing a make serve, things didn’t look quite right:

127.0.0.1 - - [21/Jan/2014 21:41:51] "GET /theme/css/pygment.css HTTP/1.1" 200 -
WARNING:root:Unable to find /theme/css/pygment.css file.
WARNING:root:Unable to find /theme/css/pygment.css.html file.
WARNING:root:Unable to find /theme/css/pygment.css/index.html file.
127.0.0.1 - - [21/Jan/2014 21:42:24] "GET /blog/2014/01/21/upgrading-pelican-3-3/ HTTP/1.1" 200 -
WARNING:root:Unable to find /blog/2014/01/21/upgrading-pelican-3-3/ file.
WARNING:root:Unable to find /blog/2014/01/21/upgrading-pelican-3-3/.html file.
127.0.0.1 - - [21/Jan/2014 21:42:24] "GET /blog/2014/01/21/upgrading-pelican-3-3/ HTTP/1.1" 200 -
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 47703)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 704, in finish
    self.wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------

So there’s the logging thing for starters. Seems one of the Pelican 3.3 devserver “improvements” is to output 3 “Unable to find” log messages for each file it serves, even if the file is actually found and served successfully. While initially confusing, it’s only a superficial annoyance.

The “broken pipe” errors were more concerning though, if only because it wasn’t clear whether they were being caused by something messed up in my system’s configuration, or they indicated a bug somewhere in Pelican.

Initially, despite the broken pipe errors, it looked like the pages were still being served correctly. Not always, as I soon found out.

Because I was hacking away at my theme, I was not only using the development server quite a bit, but also scrolling to specific parts of pages (like the bottom) to see how things were looking as I made modifications to the theme. I soon noticed things were definitely not looking right, for reasons other than my limited design skills:

That screenshot shows what should be the bottom of the page. Obviously the HTTP headers should not be there, but neither should the truncated 2nd copy of the page that appears after the HTTP headers.

I noticed these two things — the broken pipe errors and the HTTP headers in the web page — were happening frequently, but not on every page view, so it was one of those annoying intermittent problems.

I googled and found some reports of similar problems, but no where near as many as I’d expected, given that I was experiencing this really often.

Was it something in my setup?

I had a look at the Pelican source on GitHub, and sure enough, there was new stuff added to devserver in version 3.3.

Before version 3.3, the Pelican development server was basically just the standard Python SimpleHTTPServer. But in version 3.3, a small wrapper was added around SimpleHTTPServer to provide some extra functionality. The extra functionality was some basic nginx-style URL re-writing rules, to provide the illusion of “clean” URLs, without actually having to configure clean URLs the “Pelican way” (i.e. using the *_URL and *_SAVE_AS settings so that content ends up in a directory containing an index.html).

Personally I was quite happy with having clean URLs as directories. It seems a bit hacky at first, but works well in practice, and keeps things appropriately simple. While I can see how this new devserver functionality might be occasionally useful, I would question the need to make this the default behaviour, especially since (AFAIK) it’s entirely undocumented.

After a bit of investigation and poking around, I came to the conclusion that both problems I was experiencing with devserver were the result of a bug introduced in the version 3.3 changes to server.py. (And not anything in my setup.) Somewhat ironically, in an attempt to provide “fake” clean URLs, those changes broke handling of “proper” clean URLs (i.e. serving of all URLs that resolve to an actual directory on the server).

I suspect the reason there weren’t as many reports of these problems as I expected was because they are more likely to occur if Pelican is configured to generate “proper” clean URLs. This is something I’ve tried to do consistently in my configuration, but it seems I must be in the minority. The fact they’re intermittent and not exactly show stopping (though annoying/confusing) is no doubt also a factor.

In any case the whole thing bugged me enough that I decided to do a bit of tangential tinkering to try and fix the problem. (BTW, anyone out there know why SimpleHTTPRequestHandler is not a new style class under Python 2.7?)

Workaround

For anyone also regularly experiencing broken pipe errors with the Pelican 3.3 devserver and/or mangled HTML being served, there are a few ways to work around this. Obviously you could just incorporate the patch into your own setup.

But for those who don’t need the new stuff added in 3.3 (and I imagine very few people would), there are simpler options.

One is just to use SimpleHTTPServer on the command line:

cd output; python -m SimpleHTTPServer

Another, for those who want to keep using the devserver, is to change one line in Pelican’s server.py file. If you’re using a virtualenv (and if not, you should be), this file will be located somewhere like this:

~/path/to/virtualenv/lib/python2.7/site-packages/pelican/server.py

Edit server.py, replacing this line:

Handler = ComplexHTTPRequestHandler

With this:

Handler = srvmod.SimpleHTTPRequestHandler

This essentially disables the new new Pelican 3.3 devserver functionality (with the buggy bits) so you essentially get the same behavior as Pelican 3.2 — plain old SimpleHTTPRequestHandler. The makefile serve and devserver targets should now work properly.

Configuration changes for Pelican 3.3

There weren’t many changes to pelicanconf.py required after upgrading to 3.3.

The only significant change was as a result of FILES_TO_COPY being deprecated. It has been replaced with STATIC_PATHS and EXTRA_PATH_METADATA.

So in pelicanconf.py this:

STATIC_PATHS = ['images', ]

FILES_TO_COPY = (('other/robots.txt', 'robots.txt'),
                 ('other/favicon.ico', 'favicon.ico'),
                 ('other/arbitrary.html', 'arbitrary.html'),
                 )

Became this:

STATIC_PATHS = ['images',
                'other/robots.txt',
                'other/favicon.ico',
                'other/arbitrary.html',
                ]

EXTRA_PATH_METADATA = {
    'other/robots.txt': {'path': 'robots.txt'},
    'other/favicon.ico': {'path': 'favicon.ico'},
    'other/arbitrary.html': {'path': 'arbitrary.html'},
    }

The official docs don’t go into a huge amount of detail about EXTRA_PATH_METADATA, and I’m not sure if I’ve got things configured quite right. In the above excerpts, arbitrary.html represents a few arbitrary HTML files I want Pelican to copy to the root output directory. I want them copied unmodified, that’s it. But for reasons I don’t understand Pelican seems to be looking at the contents, possibly wanting to process them in some way, because I keep getting errors like this:

ERROR: Skipping other/arbitrary.html: could not find information about 'title'

I didn’t see anything like this in version 3.2, and I’m not sure what it’s complaining about. Sure, those arbitrary HTML files in question don’t actually have a title — they are just HTML fragments, mostly for use by tools and not for human/browser consumption. I just want the files copied. And they are, in fact, being copied to where I want them, so in that respect things are working. I’m just not sure why Pelican considers this an “error”.

UPDATE: I eventually found a solution to this.

Images and internal links

It seems another side-effect of the changes to STATIC_PATHS was the default location of image files changed, with the static directory being removed. This meant I had to modify any places in my theme that directly (hackily?) referenced images in the output location.

So stuff like this:

<img src="/static/images/imagefilename.png" alt="">

Became this:

<img src="/images/imagefilename.png" alt="an image">

Another change introduced in Pelican 3.3 with regards to images is the syntax for links to internal content. While the old syntax is still supported, the new syntax uses curly braces:

{filename}path/to/file

Also, as of version 3.3, Pelican gives warnings for images without alt attributes:

WARNING: Empty alt attribute for image

I don’t have many posts with image links, so it was easy enough to find the few image links I did have and add alt attributes to them, like so (assuming reST format):

.. image:: {filename}/images/imagefilename.png
   :alt: alternative text

Since I was going through my previous posts and adding alt attributes, I also updated the links to use the new internal linking syntax while I was at it.

That got rid of most most of the “Empty alt attribute” warnings, but a few still remained. Turns out Pelican issues this warning not only for images referenced the “proper” way, but also for any <img> tags included in raw HTML blocks (using the reStructuredText “raw” directive). This kind of surprised me — and seems overly protective of Pelican. Not sure how I fee about Pelican judging me on the quality of my raw (and hacky) HTML code!

Although it’s not really my code. Most of the raw HTML blocks were used to incorporate HTML/Javascript code snippets from the various increasingly ubiquitous third party services. Some of these code blocks had IMG tags with alt attributes set to an empty string:

<img alt=""... />

Turns out this empty alt attribute is what Pelican was complaining about.

Surprisingly though, removing those empty alt attribute removed the warnings. Seems Pelican is fine with IMG tags in raw HTML blocks without alt attributes, but it won’t tolerate missing alt attributes on reST image directives. Weird.

So to get a “neat” build without warning/errors I had to make sure all image directives had an alt tag, but had to remove alt tags from IMG elements in raw HTML blocks.

Success

After all that however, with a fixed devserver and image references appropriately tweaked, I did manage to get a functional, updated pelican setup, along with a mostly error/warning free build of my blog.

With that out of the way, it was now time to go ahead with updating my blog to use the latest versions Bootstrap and Font Awesome. Or at least find creative ways to procrastinate.

But that’s possibly a topic for another blog post.

Raiders of the Lost Django Registration Templates

2014-01-15T00:00:00+11:00

Psst!

Hey you.

Yes, you there, looking at the screen with a combination of confusion, surprise and frustration. Is this the first time you’re trying to create a Django website with user sign up and log in capabilities? Looking for a “built-in” way to do that?

Well, there isn’t one. Sometimes implementing common website functionality that you’d think should be easy and straightforward isn’t so much. But hey, that’s pretty much always going to be the case with any web framework. Though as always, be sure to keep in mind that despite moments of confusion or frustration with the Django learning curve, at least it’s not PHP. So there’s that.

So yes, Django has user authentication built in, and you get a nice admin interface for free if you want it, but none of that is going to help you much with user sign up, as such. You’ll either have to roll your own solution on top of the bits Django does provide, or use one of many external packages, each with their unique combination of advantages and disadvantages.

The closest thing to a “standard” way to provide user sign up is django-registration. This only does local sign up (i.e. based on username/email). If you want log in via social media accounts, that’s a whole other thing.

But if all you need/want is local sign up, django-registration will get the job done. After a bit of fiddling. Including creating your own templates.

UPDATE: Since this post was written, django-registration went through a period where it was not maintained and seemingly abandoned, which led to the creation of a fork called django-registration-redux. AFAIK both are now maintained and compatible with recent versions of Django. I have no advice to offer as to which is preferable theses days, or whether other alternatives entirely are best considered.

Keep in mind user sign up is something where there isn’t likely to be one-size-fits-all approach that will work for everyone. Different websites are likely to want different approaches, hence the different possible registration backends django-registration provides. Then there’s front end differences. Not just appearance, but also differences in front-end frameworks, and any specific django template libraries used within the project.

So this post isn’t intended to be authoritative, or an opinion as to best practices — just a starting point. Because having at least some working sample code is often useful when starting out.

What follows are some pointers and examples that might be useful for those trying to get their head around the whole process of integrating django-registration, and creating the required templates.

Required Templates

The official documentation does a pretty good job of walking through the steps necessary to install and configure django-registration.

As it points out, django-registration requires these templates:

registration_form.html
registration_complete.html
activate.html
activation_complete.html
activation_email_subject.txt
activation_email.txt

The above templates cover functionality provided by django-registration.

However, in addition to the above, most non-trivial websites would want to implement their own templates for things like login, logout and password resets. This functionality is provided by Django’s built-in user authentication system. The authentication views make use of the following templates (by default):

login.html
logout.html
password_change_form.html
password_change_done.html
password_reset_form.html
password_reset_email.html
password_reset_subject.txt
password_reset_done.html
password_reset_confirm.html
password_reset_complete.html

Example django-registration Templates

Here are some simple examples of the templates django-registration requires, using Bootstrap 3 with the django-bootstrap3 template library.

The complete code is on github.

Templates need to be placed in a subdirectory called registration somewhere where the template loader can find it. (Configure TEMPLATE_DIRS in your settings file as appropriate.)

The following examples assume the default backend is being used, with the default URLs.

`registration_form.html`

This is the template for the sign up screen. It will contain the registration form for the user to fill out.

You will also mostly likely need to provide a link to URL of the registration view somewhere in your website — for example, a “register” or “sign up” button on the home page. With the default backend the view class is registration.backends.default.views.RegistrationView, and the URL is named registration_register.

So a “Register” button on the home page will look something like this, in the home page template:

<a class="btn btn-lg btn-primary"
   href="{% url 'registration_register' %}">Register</a>

When the user clicks this link, the registration_form.html will be displayed.

A form context is passed to registration_form.html, which it will need to render. For the default backend, this will be an instance of registration.forms.RegistrationForm.

{% extends "base.html" %}
{% load bootstrap3 %}

{% block title %}Registration{% endblock %}

{% block content %}
<div class="container">
  <div class="row">
    <div class="col-sm-offset-2 col-sm-10">
      <h1>Sign up</h1>
      <p>Already registered?
        <a href="{% url 'django.contrib.auth.views.login' %}">Sign in here.</a></p>
    </div>
  </div>

  <form action="{% url 'registration_register' %}"
        method="post" role="form" class="form-horizontal">
    {% csrf_token %}
    {% bootstrap_form form layout='horizontal' %}
    {% buttons %}
    <div class="col-sm-offset-2 col-sm-10">
      <button type="submit" class="btn btn-primary">
        {% bootstrap_icon "star" %} Sign Me Up!
      </button>
    </div>
    {% endbuttons %}
  </form>
</div>
{% endblock %}

`registration_complete.html`

Confirmation to the user that registration has completed successfully.

{% extends "base.html" %}

{% block title %}Registration complete{% endblock %}

{% block content %}
<div class="container">
  <div class="row">
    <div class="col-md-8">
      <h1>Registration complete!</h1>

      <p>An activation email has been sent.</p>
    </div>
  </div>
</div>
{% endblock %}

`activate.html`

Displayed if account activation fails. The activation_key context is provided to display the failed activation key used.

{% extends "base.html" %}

{% block title %}Account activation error{% endblock %}

{% block content %}
<div class="container">
  <div class="row">
    <h1>Account activation error</h1>
    <div class="alert alert-danger col-md-9">
      <p>The activation key <strong>{{ activation_key }}</strong> is invalid or expired.</p>
    </div>
  </div>
</div>

{% endblock %}

`activation_complete.html`

Confirmation to the user that registration has completed successfully.

{% extends "base.html" %}

{% block title %}Account activation complete{% endblock %}

{% block content %}
<div class="container">
  <h1>Success!</h1>
  <p class="lead">Account successfully activated!</p>
  <p>You can now sign in to the website.</p>

  <p><a href="{% url 'django.contrib.auth.views.login' %}"
        class="btn btn-success"
        role="button">Sign in</a>
  </p>
</div>
{% endblock %}

`activation_email_subject.txt`

Subject line for the activation email:

{{ site.name }}: Please activate your account.

`activation_email.txt`

Message body of the activiation email. At the very least will need to display a link to the activation URL, which for the default backend is named registration_activate, passing through the activation key available in the activation_key context. Optionally, you can also include how many days before the activation key will expire:

To activate your account, please visit the following page:

http://{{ site.domain }}{% url 'registration_activate' activation_key %}

This page will expire in {{ expiration_days }} day{{ expiration_days|pluralize }}.

Django 1.6 issues

Note that changes to the auth module introducted in Django 1.6 cause some incompatability with django-registration when it comes to password URLs. Until django-registration is updated you may need to add a workaround to your URLs if you’re providing custom password templates:

url(r'^accounts/', include('registration.backends.default.urls')),
url(r'^password/change/$',
                auth_views.password_change,
                name='password_change'),
url(r'^password/change/done/$',
                auth_views.password_change_done,
                name='password_change_done'),
url(r'^password/reset/$',
                auth_views.password_reset,
                name='password_reset'),
url(r'^accounts/password/reset/done/$',
                auth_views.password_reset_done,
                name='password_reset_done'),
url(r'^password/reset/complete/$',
                auth_views.password_reset_complete,
                name='password_reset_complete'),
url(r'^password/reset/confirm/(?P<uidb64>[0-9A-Za-z_\-]+)/(?P<token>.+)/$',
                auth_views.password_reset_confirm,
                name='password_reset_confirm'),

Demo code

The above templates, as well as complete set of Bootstrap-friendly sample registration/authentication templates, are on github. There’s also a simple but functional demo site to show how everything fits together.

Booze To The Node

2014-01-06T00:00:00+11:00

Lately, it seems that when it comes to politics and technology, technical merits don’t matter. Politicians only seem to care about technology to the extent it allows them to distribute pictures of their privates.

So if we’re trying to engage our political masters on issues important to us, why should we constrain ourselves with such things as logic and reason, if our politicians don’t?

Perhaps we should take a leaf from the sordid playbooks of politics and advertising. Perhaps we should pay less attention to such things as logic and facts, and more to spin and hyperbole.

Because it’s not as if we’re going to persuade an apathetic public to care about our abstract technical issues with mysterious acronyms and jargon. Whether we’re trying to persuade politicians or the public, we need to work out a way to communicate that is effective for an uncaring and non-technical audience.

With that in mind, here’s my attempt to explain — for a non-technical audience — the differences between Labor’s superior Fibre to the Premises (FTTP) NBN, and the Coalition’s crappy Fibre to the Node (FTTN) alternative.

Why Fibre to the Node is stupid: a non-technical guide for non-geeks and alcoholics

Let’s say you want to get drunk. Really, really drunk. Because your ADSL doesn’t work. And Telstra won’t fix it. And even your analogue phone line is a crackly mess of static because the copper in your street has degraded to buggery. And Telstra won’t fix it. So you have decided to get drunk. Really drunk.

Not that I’m condoning alcohol consumption as a way of dealing with ADSL problems. But let’s say we’re living in an alternative fantasy universe where Australians drink responsibly. (Because only in an alternative fantasy universe would Fibre to the Node be an acceptable technical solution to our outdated, half-assed, patchwork broadband infrastructure.) An alternative universe where boat people are not an issue, because any boat people that do get here are so shocked by our crappy internet that they fuck off to countries with decent upload speeds, scoffing at us primitive Australians still using copper networks for internet connections.

So you’re at home and you want to get drunk. But you’re out of booze. And it’s a long way to the nearest bar or bottle shop. And you don’t have a car. And public transport doesn’t work because the Liberals are in power.

Imagine a Booze to the Premises system, where you can get any kind of alcoholic beverage — and as much as you want — delivered right to your home almost instantly, 24 hours a day, no matter where you live. Pretty great, right? So that’s the idea of universal FTTP technology. It will allow us to quench our thirst for data downloads quickly and efficiently — no matter what time, day or night, or what the weather is like.

So ideally, if you’re really, really thirsty, you’d want Booze to the Premises. But what if you were stuck with Booze to the Node?

Booze to the Node also promises delivery of alcoholic beverages, but it’s much like the promise of a degenerate alcoholic — forgotten the next day. Deliveries takes much longer — sometimes ridiculously long — and sometimes they don’t arrive at all.

What’s worse though, is that your booze won’t be delivered to your home — it can only be delivered to a big ugly box at the end of the street. A big box with lots of lights and electrical equipment inside that uses huge amounts of electricity and therefore contributes to greenhouse gas emissions.

Worse, everyone in the area has to get their booze delivered to the same box. So there is a limit to how much you can order at once. The more people getting deliveries, the less each person can have delivered in one go. And once your order arrives at the big ugly box at the end of the street (if it does arrive) you have to walk down the street to get it.

“But that sounds silly!” you say. “Surely, if they can deliver booze to the end of my street, then wouldn’t it just be better and easier and more practical all round to go that little bit further and actually deliver it right to my home?”

Exactly!

What could have been an efficient system of getting drunk on conveniently home-delivered booze in the comfort and safety of your own home, turns into a time consuming walk down the road, only have to carry all those heavy bottles back home yourself, and all at the mercy of the weather. If it’s raining, things will get slippery and packaging will soak through, making it likely at least some bottles will get dropped and break on the ground. All that expense and effort and nothing to show for it! What a waste! You’ll then have no choice but to order more booze and go through the whole process again!

But it gets worse. In addition to the inconvenience of having to walk down the road to pick up your order, the trip itself is likely to be quite unpleasant, because the streets are full of cranky ADSL users. They are angry and driven to the brink of insanity because their internet connections keeps dropping out. It seems no one can do anything for them. Telstra certainly won’t do anything. And even if there was someone to call for help, it unlikely the phones will be working because the phones use Telstra’s copper network, which Telstra won’t fix.

So these disgruntled ADSL users just roam the street, plaintively wailing for help, asking strangers if their ADSL is working and frantically questioning them about their distance from the exchange. Sadly, there is no choice but to venture out and try to avoid them. They will get in the way and slow things down, and if there’s lots of them around at one time you may have to go a different way (even if it’s longer), or just wait till another time. But these angry ADSL users are essentially harmless. Just ignore them and hope they go away. That’s what Telstra does.

Clearly, Booze to the Premises is the vastly superior option, and it’s what the majority of Australians would prefer. But it’s not what we’re going to get.

The Coalition is in power now, so much like a devious con-artist, they’re trying to get us to accept Booze to the Node.

They’re trying to screw us.

Being screwed is bad enough. What’s worse is that we are getting screwed by egotistical, hypocritical politicians who seem to think the internet exists mainly so that they can distribute pictures of their genitals.

There must be some way we can distract our politicians from taking photos of their privates long enough to explain our desire for a modern, reliable broadband network? There must be some way to make them understand that we want our internet the way we want our booze: reliably supplied, readily available, whenever we want, wherever we happen to be living, and in quantities suitable for binging.

Anything less would be un-Australian.

Screwed To The Node

2014-01-05T00:00:00+11:00

This whole NBN thing has got me thinking about what happens when the world of technology intersects with the strange world of politics. When geeks go up against politicians, the geeks inevitably get screwed. Which is exactly what is happening in Australia with the National Broadband Network, now that the Coalition is in power and determined to undermine Labor’s visionary Fibre to the Premises (FTTP) network with a crappy Fibre to the Node (FTTN) alternative.

Much has been written to explain that Labor’s FTTP plan is superior to the Liberal’s FTTN version, by people quite knowledgeable on these matters. Which is all well and good, as long as we don’t work under any misapprehension that things like logic, facts, and technical details matter in any way shape or form when it comes to politics.

Politics is a world seemingly untroubled by cognitive dissonance, heedless of hypocrisy, and unconcerned with facts. Which means we end up with the sort of government where the head of a parliamentary ethics committee — and married Catholic — is caught sending pictures of his penis placed in a glass of red wine to his mistress. We end up with the sort of government where the Minister for Women is an out of touch misogynist deeply unpopular with women.

We end up with the sort of government that racks up debt while negotiating secret trade agreements, that undermine Australia’s sovereignty, after hyperbolicaly complaining while in opposition about budget emergencies, and insisting that poor people in leaky boats didn’t deserve compasion because they were a threat to Australia’s sovereignty.

It’s a sad indictment on our lobbying ability that we are losing even to politicians unable to do their job effectively because they’re too distracted taking pictures of their junk with their smartphones — and sexting with mistresses.

Do we really think we can reason with people so unreasonable? Do we really think we can change their minds with logic, when their actions lack any logic whatsoever? (Or common sense, or basic decency for that matter!) Do we think we can change their minds with facts or technical details, when they revel in their ignorance? I don’t think so.

Since facts, logic and reason count for little in politics, a strategy relying on any combination of them is unlikely to succeed. Especially in a political landscape where hypocritical, hyperbolic bullshit reigns supreme. Which is one of the reasons the technical community constantly get screwed by those we elect to supposedly represent us — because we seem to be unable to out-bullshit the bullshitters. Instead we misguidedly try to reason with them.

So maybe we should try something else. Something less logical. Maybe we should become as unreasonable as the politicians we are vainly trying to reason with. The politicians we are making bad political decisions that affect us.

After all, why should we be constrained by facts, logic and reason, if our politicians aren’t? If they’re going to resort to hyperbolic bullshit, then why shouldn’t we?

This gives me an idea. Continued in part 2…

The Deviant Luddites Screwing Us

2013-12-30T00:00:00+11:00

Screaming into the Broadband Void (Part 4)

It’s a great day if you’re a late night talk-show…you know…thing…this…because…former congressman Anthony Weiner is returning to politics! Listen, if you had to do this crap every day you would love it when you hear a weiner is running. Weiner running again? That takes balls! Then you could say things like: “I guess you can’t keep a good weiner down.” … Now Weiner is a Democrat, but from what I’ve seen, sometimes he leans to the right.

—Craig Ferguson, The Late Late Show, April 2013

Here in Australia, those of us with an interest in technology — us geeks — are currently undergoing a gangbang sized screwing on two issues in Australian politics that potentially have profound implications for the whole country, but especially for anyone who works in IT and other technology industries: The National Broadband Network, and the Trans-Pacific Partnership Agreement

The Coalition is giving up a historic opportunity to correct some of the problems they caused by privatising Telstra without structural separation. Instead of continuing what Labor started and building a modern Fibre to the Premises broadband network, they will doom Australia to a future as a technological backwater with their inferior Fibre to the Node alternative. Although perhaps that’s just as well, because if the intellectual property provisions of the Trans-Pacific Partnership are implemented as currently drafted, doing just about anything useful on the internet will become illegal! Even innocuous web browsing will become a crime! Yes, they’re even trying to make web browsers illegal.

The TPP is an odious attempt by multi-national corporations to control the way we use the internet and consume media content (ebooks, music, movies etc). It’s an attempt to eliminate all kinds of common consumer behaviour — like transferring legitimately purchased media from one legitimately purchased device to another for more convenient consumption of that media. The TPP is, amongst other things, an attempt to ensure you may only consume the media you purchase in the way the corporation you purchased it from wants you to consume it, regardless of convenience. It’s an attempt to make sure you can only use their proprietary software with their proprietary hardware and their proprietary copy protection — all of it incompatible with those of their competitors.

Welcome to the future. Only in a world where using the internet is mostly illegal does having a crappy Fibre to the Node network make any kind of sense.

That’s the sort of future our wonderfully competent elected leaders are envisioning for us.

As the NBN rollout continues — albeit Malcolm Turnbull’s crappy, crippled FTTN version — and as node cabinets start appearing on street corners, each one will represent a failure of us in the technology industries to make a difference in the political process. Each one will be a monument to our geeky failures, and further proof of the uselessness of slacktivism. Each one will be a reminder of how effectively our political overlords are screwing us.

However, as the year comes to an end, if we look back at some of the things our beloved political masters did that made headlines, it certainly makes one wonder about the calibre of our esteemed elected leaders. It makes one question their character, and reflect on their shortcomings. Yet, these are the sorts of people who are so efficiently screwing us? Once again, this year showed that just when we thought that the dignity of public office couldn’t be disgraced any further, our beloved elected officials proved there were always new depths of disgrace to plumb.

Canada provided comedians around the world with the gift of Rob Ford, Toronto’s crack-smoking mayor. But he only smoked crack while in a drunken stupor, so that’s OK, apparently, in Canada.

In the US, former San Diego Mayor and purveyor of creepy smiles, Bob Filner, was sentenced for sexual harassment. Disgraced former New York Governor and call girl aficionado Eliot Spitzer tried and failed to become New York City Comptroller.

Even renown sexter Anthony Weiner attempted an apologetic political comeback. He was even leading in the polls early on, before his campaign was ultimately derailed by another sexting scandal, proving that Weiner (AKA “Carlos Danger”) really is the comedy gift that keeps on giving.

The Spitzer and Weiner comeback campaigns make me wonder about the people who dedicated resources (money, time, manpower) to help these individuals get elected. I’m all for second chances, but is the talent pool really so shallow that there weren’t any worthwhile, non-sexting, non-call-girl-choking candidates to whom those resources could have been better allocated? If nothing else, in Weiner’s case, it showed he had learnt nothing the first time round, and was certainly not worthy of second chances.

Here in Australia we had our own Weineresque sexting scandal, when revelations surfaced that Queensland MP Peter Dowling had taken pictures of his penis placed in a glass of red wine and sent them to his mistress. This from a married conservative family man, a supposedly devout Catholic, and chair of the parliamentary ethics committee!

I wonder if there was any point in time, before the media got hold of the story, that Peter Dowling realised the irony and blatant hypocrisy of what he was doing? I wonder if just before he was about to send a dick pic to his mistress, he ever paused for a moment, to ponder if maybe this would at some point in the future come back to haunt him? I wonder if, during his liaisons with his mistress, it ever occurred to him that, as a conservative family values politician and chairman of the parliament’s ethics committee, perhaps screwing around on his wife wasn’t the best career move, not to mention morally questionable?

I suspect not. There seems to be something about the ego and lack of self awareness of politicians that engenders such blatant tone-deaf hypocrisy. It’s the same runaway ego and lack of self-awareness that results in Tony Abbott appointing himself minister for women, most likely completely oblivious to how obnoxious the whole idea appears to an electorate that sees him as misogynistic and out of touch with women.

Just like it’s the same level of misguided yet egotistical stubbornness that enables Malcolm Turnbull to advocate for a Fibre to the Node broadband network in the face of ridicule and criticism from the entire technical community.

It would no doubt be an understatement to suggest that a family values advocating leader of a parliamentary ethics committee taking pictures of his penis in a glass of wine and sending them to his mistress — perhaps not even considering the possibility of repercussions — probably has his fair share of personality flaws.

I suspect it’s the same kind of personality flaws that enable Malcolm Turnbull to push forward with his silly plans for a Fibre to the Node broadband network, ensconced in the comforting self-delusion that all the experts deriding FTTN technology are wrong, and that there won’t be any negative repercussions resulting from his choice of inferior networking technology. I’m sure Malcolm believes no one will notice the slow speeds and dropouts — just like no one currently notices the slow speeds and dropouts of ADSL.

Politics is a strange job, and it attracts some strange people, often for all the wrong reasons. It’s pretty obvious by now that our politicians are generally deeply flawed individuals. It might even be their flaws that help them succeed in the strange world of politics.

I think this might explain why us geeks constantly get screwed over by these dick pick distributing deviants. There is a huge disparity in personality types and abilities between the world of politics and the world of technology. They require vastly different skills and personality traits to succeed. That might be partly why our political leaders make such incredibly bad decisions on matter relating to technology — they don’t understand us. And it’s also partly why we don’t fight back effectively — because we don’t understand them. We tend to naively think that we can reason with them. That we can change their minds by explaining the technical merit of our position. So we blog and tweet and sign online petitions and post in forums — all of it pointlessly screaming into the digital void. Because politicians don’t care about any of that. None of that directly affects them in their quest for whatever emotional validation they seek by trying to attain public office. None of that directly affects their quest for personal gain, power, ego and status — so they don’t care.

Which means to make them care about our geeky concerns, we probably need to learn to think a bit like our deviant political overlords — as disturbing and distasteful as that prospect may be. Because surely their flaws and seeming lack of self control could provide some sort of advantage for us? Since our political leaders seem incapable of anything except taking pictures of their genitals and distributing them to unsuspecting women on social media, isn’t there any way we can take advantage of this?

Because it’s bad enough we’re being screwed, but it’s worse that we’re being screwed by these incompetent, power hungry, sex crazed egotists with no self control and and even less self awareness. These are the people who are screwing us? These dick pic distributing deviants? These family values campaigners by day, and adulterous whore mongers by night?

We tend to think we’re kinda smart, aren’t we? So is there no way we can outsmart our politically meddlesome masters? Is there no way we can take advantage of their character flaws to get a good outcome on technology issues we care about? (And leave them to destroy their own careers in entertainingly embarrassing ways the rest of the time?)

While our salacious political leaders are busy cavorting with prostitutes and taking pictures of their cocks, can’t we somehow work out a way to get decent broadband? While they’re distracted with finding novel ways to destroy their reputations, can’t we work out ways to preserve our online freedoms? While they’re sticking their dicks in glasses of wine, can’t we work out ways to reform the patent system?

Who knows, maybe somehow, in this process, if our political leaders realise we’re watching them closely, ready to use any of their weaknesses or indiscretions against them, maybe one day there might come a time where some dignity is restored to public office.

At least until the next sexting scandal.

Hacking Government: An Ineffective Digital Militia

2013-12-27T00:00:00+11:00

Digitally Pissing in the Wind (Part 3)

“I do think the pace of change in the world is increasing. Part of what I would think about is, I think that we haven’t adapted mechanisms to deal with some of our old institutions like the law and so on aren’t keeping up with the rate of change that we’ve caused through technology. You know, if you look different kinds of laws we make, and things like that, they’re very old. …The other thing in my mind is we also haven’t maybe built mechanisms to allow experimentation. There’s many, many exciting and important things you could do that you just can’t do ’cause they’re illegal or they’re not allowed by regulation. And that makes sense, we don’t want our world to change too fast. But maybe we should set aside some small part of the world, you know, like going to Burning Man, for example. Which I’m sure many of you have been to. Yeah, a few Burners out there. That’s an environment where people try out different things, but not everybody has to go. And I think that’s a great thing, too. I think as technologists we should have some safe places where we can try out some new things and figure out: What is the effect on society? What’s the effect on people? Without having to deploy it into the normal world. And people who like those kinds of things can go there and experience that. And we don’t have mechanisms for that. So those are the kind of things I would think about.”

—Larry Page, Google I/O 2013 keynote

The lack of serious gun control in the US is a testament to how effective the NRA and gun lobbyists are at stopping the government from encroaching on their gun rights. By contrast, things like a fundamentally broken patent system and legislation like the DMCA are a testament to how useless geeks are at stopping the government from encroaching on their digital rights.

It’s strange to observe the gun control debate occurring in the US. Especially living half a world away, in a country that has had very strict gun control laws for many years, an no more mass shootings as a result. It’s strange hear the invective used by the gun lobby about gun control laws being akin to government tyranny, when here in Australia things don’t seem any more tyrannical than any other alleged democracy. It’s easy to make fun of Americans and their love of guns (though not if there are any in the room, because they’d probably be armed), and it’s easy to mock the NRA and their ridiculously overwrought and hyperbolic stance on gun control. Especially if one’s main source of overseas news is The Daily Show.

Yet, even though the issue of gun control is not one that I have much personal interest in, I have been thinking a bit about the NRA because of two current issues in Australian politics that have no (obvious) connection with guns — the newly elected Coalition government’s vastly inferior Fibre To The Node plan for the National Broadband Network, and The Trans-Pacific Partnership Agreement.

Both are just two recent examples of something that’s been happening over and over again for long time now, all over the world — politicians doing really stupid things and making really bad decisions when it comes to matters of technology, while those most affected do little to effectively fight back. It’s politicians and other Merchants of Bullshit in power screwing the geeks — and winning. Of course, with the way technology is now a part of everyday life for most people, this means it’s not just the geeks getting screwed — it’s pretty much everyone that uses the internet in any way, shape or form. In the case of the TPP, the effects of the technologically toxic intellectual property provisions will affect just about everyone. As Julian Assange eloquently put it:

“If you read, write, publish, think, listen, dance, sing or invent; if you farm or consume food; if you’re ill now or might one day be ill, the TPP has you in its crosshairs.”

Modern democracy has wandered far from any ideals of government by the people and for the people. The TPP is a great example of government by the rich elite for multinational corporations — as large corporations are the only beneficiaries of the IP changes, and everyone else is worse off.

I find that above quote from Larry Page kind of disturbing, because it seems to imply that even a company with the money, power and influence of Google doesn’t know how to hack government. Even Google doesn’t seem to know how to deal with technologically clueless yet meddlesome politicians who often don’t understand the repercussions of their actions. This is in stark contrast to the NRA and the gun lobby, who seem to have a remarkable ability to get politicians to do what they want, even if what they want is kind of insane. While it might be hard to read into the thinking behind Larry Page’s rambling Google I/O 2013 keynote speech, a lot of it does sound like a geek frustrated at being screwed by government. But while Page might be imagining a fanciful geek utopia unrestrained by burdensome, restrictive and misguided government intrusion, I’m wondering where are the geek equivalents of Wayne LaPierre, the NRA’s executive vice president — a guy with an uncanny ability to keep government at bay in the face of much media ridicule and public hostility. Where are the tech industry’s rhetorical soldiers in the war against government overreach?

If it’s too hyperbolic to describe it as a war, then it is at the very least a game. It’s a game we have to play whether we like it or not, no matter how distasteful or uninteresting it may be, if we want to have any chance protecting our digital rights and freedoms.

The NRA appears to be an organisation that’s remarkably effective at playing the dirty game of politics, even though it’s far more concerned with the interests of its corporate backer in the firearms industry than the rights of individual gun owners.

Even though I care little for their cause, I find myself fascinated by their methods. Like them or loathe them, they are obviously effective. Much like money printing economists, the gun lobby is very good at getting politicians to do insane things, like letting insane people easily buy guns. While economists have managed to convince the nation that the cure for an economy crippled by too much debt is more debt, the gun lobby have managed to convince the nation that the cure for a country plagued by gun violence is more guns.

Both economists and the gun lobby have demonstrated the remarkable power of bullshitting to achieve their insane ends. So maybe there’s something us geeks can learn from all this, as we struggle to battle governments more and more determined to encroach on our technology industries, and take away our digital rights and freedoms.

Maybe there is some truth in the overblown rhetoric of the NRA. Maybe in the modern bullshit world of politics corrupted by mega-corporations with their mega-campaign-donations, there is no such thing as government without government overreach. Because there will always be an agenda.

Perhaps the gun lobby is right to fear a little gun control will lead to a lot of gun control. After all, it started with a little software patent sneaking it’s way into the patent system, and now decades later, stuffed full of junk software patents, the patent system is a horrible, broken, bloated mess that stifles innovation and benefits no one apart from patent trolls and large corporations.

Perhaps the gun lobby is right to fear a gun owner database, as a little government data collection will inevitably lead to lots of government data collection. Just like limited government power to intercept communications of criminals and select terror suspects has lead to the NSA building a multi-billion dollar data centre so they can read and store every email from everyone everywhere forever!

So maybe the solution to overzealous government sticking its tentacles where they don’t belong, is to fight bullshit with bullshit. So maybe the gun advocates have got the right idea. They are not giving the government the benefit of the doubt. At all. No wriggle room, no leeway. Because maybe they have a point: That any government, given half chance, will abuse any power it is given. So they do whatever they can to stop even reasonable gun control measures, out of a perhaps a legitimate fear that reasonable gun control will inevitably lead to unreasonable gun control. So the try to ensure the government doesn’t get any more power, no matter how well meaning the intent of that power might be.

There is possibly as lesson for us geeks in all this. Maybe this is the attitude we need to adopt. After all, we have been screwed many times by government overreach. Legitimate intellectual property laws (copyright) lead to a broken intellectual property system (software patents, DRM). Small scale wiretaps lead to widespread NSA spying. Give the government an inch, and they pull the whole fucking rope out of your hands and arrest you on suspicion of terrorism! (Because who else but a terrorist would be holding a rope!) So maybe it’s time for an NRA style zero tolerance approach to government restrictions in the digital realm, complete with hyperbolic rhetoric. No software patents, because software patents are stifling innovations and destroying the technology industry. No censorship, because the government will inevitably use it to curtail free speech. No copy protection, because a little copy protection will lead to restricted competition and monopolistic practices by corporations.

It’s also instructive to note that not only is the NRA good at what it does, but it no doubt helps that gun owners tend to be a vocal and passionate bunch. This quote from an academic in a Salon.com article sums things up nicely:

“The issue is not so much how much the NRA gives any senator or member of the House, it’s how they can make their lives miserable. And how they make their lives miserable is they e-mail ’em, they call ’em, they fax ’em, they show up at meetings. The typical person who is for gun control is very different from the [pro-gun] person calling you or being right there, being an annoyance, hassling you personally. They’re much more activist than the other side and that’s what really produces their gains.”

Compare and contrast this to the pointless geeky slacktivism of opponents to the Coalition’s FTTN NBN.

We geeks bitch and moan in blogs, online forums and social media, where politicians don’t care and don’t notice. It’s time to take the battle into the real world, and really get in the face of politicians, if we want to make a difference.

Politicians are going to keep screwing us geeks until we learn how the make politicians fear us, in the same way American politicians fear the wrath of the gun lobby.

It’s time for the geeks to take a stand. It’s time to let the politicians know that we will not give up our digital rights without a real fight — in the real world. It’s time to let the politicians know they will have to pry our digital rights from our virtual cold dead digital hands.

Quantitative Geek Screwing

2013-12-19T00:00:00+11:00

Again.

The only kind of economics that is worth a damn is philosophical, not mathematical or scientific. Real economics is philosophical – like history or literature – in the sense that it merely tries to observe and understand, not to manipulate people as if they were pawns on a chessboard.

The other kind of economics – modern, “scientific” economics – is a fraud. It pretends not only to understand an economy, but also to be able to improve it.

Modern economists have been at it for nearly 100 years: planning, organizing, theorizing, controlling, adjusting, pontificating, forcing, setting prices, suppressing interest rates and committing other acts of wanton economic vandalism. During that time, there has not been a single confirmed sighting of anything close to success.

—Bill Bonner

Occasionally, I like to ponder the tough (and possibly nonsensical) questions in life. Like, why do politicians pay no attention when knowledgeable people within technology industries make logical and well thought arguments on issues that directly affect their industries? Yet, thoroughly discredited economists, who have been proven wrong many times before, still wield so much power that they can influence politicians to implement illogical policies that would have once been considered completely crazy?

What leads me to such pondering is the confluence of recent events. In particular, the global financial crisis and its aftermath; and more recently, the federal election here in Australia. As predicted, the Coalition comfortably won the election, which will likely have many implications for the technology community in Australia, but perhaps the most prominent concern is the fate of the National Broadband Network. With the Liberal Party in charge, the mostly fibre NBN as we know it is now dead.

It’s unfortunate that the issue of broadband infrastructure has become so politicised in Australia. It means that those of us involved with technology — the geeks of Australia — are going to be screwed. Again. Because based on past experience, when it comes to battling politicians, we tend to lose.

Sure, us geeks might have a minor victory, every now and then. But there are many, many more failures.

Perhaps because we are at a natural disadvantage when it comes to matters of politics. We might be smart, but this means we also tend to be logical and rational — maybe too logical and rational. Success in our careers is somewhat determined by our abilities to think logically and analytically. We are therefore prone to take that same approach to other areas. Even in areas where things like logic and reason are not valued at all, and may in fact be a disadvantage — like politics.

Which is why we might think the idea of replacing Australia’s ageing, degraded copper-based communication network with new optic fibre infrastructure would be an eminently sensible thing to do, for a whole swathe of sound technical reasons. That doesn’t mean the politicians and economists running the show will see it that way.

We might forget, or find it difficult to fathom, that in the world of politics, spin and bullshit reign supreme. Such antiquated notions like truth and facts don’t matter much when dealing with the Merchants of Bullshit. While we were learning and honing our technical skills, the politicians and lawyers and economists were busy honing their bullshitting skills. It is these master bullshitters that are now our adversaries in areas where technology and politics intersect, like the NBN, but in many other areas as well. We have no choice but to engage in battle with these bullshitters, because our worlds are likely to intersect more and more often, as new technology disrupts existing industries.

It probably doesn’t help that there’s a vein of naivety and apathy about all this within tech communities. We naively think technological solutions alone will win. We are too apathetic to engage in what, to us, is the really boring game of politics.

But just because we aren’t interested in meddling in politics, doesn’t mean politicians and others in power won’t meddle in our industries. Quite the opposite, actually.

They will meddle as they see fit, despite their lack of understanding of technical issues.

We’ve seen this clearly in Australia for decades now, when it comes to technology and communications policy. The supposedly pro-free-market Liberal Party privatised Telstra in a way that made it impossible to have true competition in broadband, because competitors could not compete with Telstra’s vertically integrated monopoly position in the market. Yet now, over 15 years later, the Liberal party is arguing that the private sector should build broadband infrastructure, not the government. Then why hasn’t the private sector done so by now? They’ve had time. Why are the majority of broadband users in Australia still relying on ADSL technology over poor quality copper wiring?

Rational arguments from the technical community are falling on deaf ears. Instead, the new government is already reframing and spinning the issue to suit their purposes. In response to the totally reasonable concern from those within the technology industry, Communication Minister Malcolm Turnbull has dismissed criticism as reckless, petitions as undemocratic, and claimed the government has a mandate to implement crappy communications policy.

Instead of taking advice from people within the technology industry who actually know what they’re talking about, decisions about the future of one of Australia’s biggest technology infrastructure projects will be entirely determined by politicians and economists.

The same professions that largely caused the global financial crisis.

OK, so not all politicians and economists are equally clueless. And when it comes to the global financial crisis, there’s plenty of blame to apportion, both to governments and the private sector. But it’s clear some seriously flawed combination of government policy, based on some seriously flawed economic thinking (assuming anyone was actually thinking), contributed to the global financial crisis. In a sane world, this means the GFC would have completely destroyed the credibility of many mainstream economists, ratings agencies, and politicians.

That doesn’t seem to have happened. The same groups of people are still in charge, the ratings agencies are still in business, and many of the same economists who advocated the market interventions that contributed to the GFC have the temerity to give advice on how to fix the economy — and people are actually listening to them.

The tech community could possibly learn something from those economists. Not about economics, clearly, but about bullshitting for influence, power and profit.

Seems to to have worked well for them. Economists still hold the same level of power and influence as before — maybe even more. They know how the game is played, and they are so good at playing the game that not even one of the greatest financial crises in history could derail their train of bullshit.

Too much debt in the economy caused a major financial collapse? I know, let’s inject trillions of dollars of more debt into the economy, because nothing could possibly go wrong with that idea!

“Printing money” has negative, hyper-inflationary connotations, and historically has never, ever worked? I know, let’s call it “quantitative easing”, and this time it’ll work for sure, because this time it’s different!

Whenever I hear the term “quantitative easing”, I can’t help thinking about a rather inconsequential controversy in Australian politics way back in the pre-financial crisis 90’s. An upstart wannabe political operative from an upstart minor political party, in response to some aggressive questioning from a journalist, said (possibly flippantly) that they could fund their poorly thought out populist and racist policies by printing money. Whether he meant that seriously, or just said it to get rid of the journalist was unclear, but the media at the time had a field day with that quote! Oh what a silly racist, they decried. Not only is this new political party xenophobic and divisive and racist, but economically illiterate as well! Everyone knows money printing is hyper-inflationary and stupid and silly and insane! Theses people are not to be taken seriously, and have no business playing the grown up game of politics! If we’re going to have politicians peddling divisive racism, they must at least be economically literate racists!

Amazing how much one global financial crisis can change the fortunes of money printing as a concept!

Now, money printing is all the rage, with the Federal Reserve leading the way, and little criticism from the mainstream media. The money printers are hailed as saviours of the global economy, as opposed to shady private banking cartels covering their own asses, and screwing the American middle-class in the process. And the financial markets love it! Everyone has bought into the bullshit so thoroughly that the merest hint that the Fed will taper sends financial markets into a panic.

It might be instructive if those in the tech community with allusions to activism kept in mind that this is world we are living in. A world where you can be proven wrong by a global financial crisis, and still retain credibility. A world of Schrodinger’s money printing, where printing money is both bad and good, depending on who is advocating it, and their agenda, not to mention how it’s branded.

A world where economists have more say over the future of Australia’s broadband network than technology experts.

Economists have managed to convince the world that to cure to all our economic ills is an obscene amount of money printing, a prospect that would have once been unthinkable in economic circles.

But the tech community can’t even convince a few politicians of the benefit of an optic fibre broadband network.

Not to mention that we also failed to stop politicians censoring the internet, spying on us, invading our privacy online, and taking away our digital rights.

Clearly, us geeks need to work on our bullshitting skills, if we have any hope of out-bullshitting the bullshitters.

National Broadband Cyber-Futility

2013-12-02T00:00:00+11:00

Digitally Pissing in the Wind (Part 1)

With the Liberal Party winning the election as expected, they now get to make good on their election promise/threat to take everything that was good and holy in Labor’s FTTP National Broadband Network and replace it with a hellishly cheap and nasty FTTN abomination. Because here in Australia, we can’t have nice things, apparently. Like decent broadband. Or a technology industry. Or science. Or women cabinet ministers. At least not when the Liberal Party is in charge.

Australia’s tentacle of every government’s favourite money-sucking vampire squid, Malcolm Turnbull, has taken over as Communications Minister, and we now get to watch with horror as Labor’s wonderfully aromatic, premium, full-bodied broadband brew gets unceremoniously flushed down the toilet, to be replaced with Turnbull’s crap-in-a-cup-and-call-it-coffee concoction.

Successive governments have screwed us on broadband, through various combination of incompetence, inaction, and ideology. Labor’s NBN was a bright spark in Australia’s dim and gloomy technology landscape. The Liberals are now going to do all they can to make sure Labor’s visionary plan for near universal FTTP broadband will be relegated to history as just a geeky cock tease — a political mirage in an otherwise barren desert of Labor Party incompetence and pointless infighting.

Now, the Liberal Party gets to screw us. Again. Which is hardly surprising, but all the more galling because this represents an opportunity for the Liberal Party to fix a problem they created. Australia’s current broadband mess is in large part a result of the Howard government privatising Telstra without structural separation. Now that they’re in office once again, they have the opportunity, after decades of inaction, to throw out the ancient copper and put Australia on the path to a modern broadband network. Instead, they’re just going to screw us all over again.

What’s really depressing though, is how this just seems to be another example of politicians, lawyers and other Merchants of Bullshit doing whatever they can to screw geeks. It’s the same pattern playing out over and over again. It’s the same pattern that’s been happening for decades now. And despite many embarrassing defeats, the geeks seem to have learned nothing about fighting back effectively.

Of course, it’s a global phenomenon, this attack on geek interests. From software patents to DRM in music files; from DMCA to region codes of DVD and Blue Ray; from the ever present threat of internet censorship to the many other subtle and not so subtle ways of stripping us of our rights and privacy online. Geeks are being screwed. It’s the bullies of the adult world telling geeks they can’t play here. It’s the established players enlisting thugs in suit with law degrees to legislate and sue the geeks into submission, put them in their place. To make it clear the geeks should fuck right off and play somewhere else, when their disruptive technology happens to encroach on someone else’s turf. In Australia, the NBN has become just another example of this.

The problem is that geeks are doing what geeks always do when they’re displeased: whining about it online. They’re blogging, and tweeting, and posting in whatever online forum they can. Which is all well and good, as long as we don’t labour under any misconception that anything we do online will make any tangible difference in the real world.

If it’s not exactly a war, it’s at least a game, of sorts. It’s us, the geeks, against them, the Merchants of Bullshit in power. And we play the game particularly badly. Because this is not a game that is played online, in blogs or social media. It’s played out in the real world. It’s played with lawyers, lobby groups, propaganda/advertising, think tanks, bribes (i.e. “campaign donations”). It’s played with spin, with communicating effectively to a possibly apathetic public, with courting the influence of those with money and/or power, and with all the tools of influence and persuasion that money and power can command. Blog posts and trending hashtags are just so much digital pissing in the wind, which will ultimately achieve nothing. As usual.

Even an online petition, no matter how successful, is unlikely to make any difference in itself, though it may have some utility in fomenting dissent. Petitions are essentially a democratic distraction — merely a mirage provided to the powerless to give them the illusion of influence, in a way least likely to actually influence anything. I doubt Turnbull and his corporate puppet masters are quaking with fear because of an online petition. Turnbull even had the balls to try to reframe the petition as undemocratic, which is an indication of the unabashed, high level bullshit we are dealing with here.

Turnbull claims the Coalition’s election victory gives them a mandate to implement crappy broadband. This conveniently ignores that the election was lost by Labor and their laughable leadership squabbles, rather than won by the Liberals with their laughing-stock Luddite leader.

Australia grudgingly elected the Liberal party with a mandate to never let either Kevin Rudd or Julia Gillard become Prime Minister ever again — and nothing more. And given our suitably low expectations of politicians, this is pretty much all we can reasonably expect the Liberals will successfully achieve while in office. Well, that, and probably some sort of sexting scandal, which seems to be disturbingly obligatory these days. No electorate in the world has ever given any politician a mandate to send pictures of their genitals to unsuspecting users of social media, yet there seems to be quite a bit of that going on around the world at the moment. Perhaps politicians are labouring under a misconception that social media exists only for them to distribute pictures of their privates.

Instructively though, some Merchants of Bullshit know how to play the game so well that even stupidly embarrassing scandals don’t always preclude disgraced degenerates from running for public office. If abject failure and public humiliation are the modern day prerequisites for power, then who do we have to send pictures of our genitals to around here to get a decent broadband network built? Which call girls must we have adulterous liaisons with before the politicians take our concerns seriously?

The reality is even if every single person in Australia signed the petition, Turnbull and the Liberals would still argue that they have a mandate to do whatever they hell they want. Because they can. Because they know they can get away with it. Because they know the worst that will happen is some snarky blogs and tweets will be written.

The online vitriol is certainly not working so far — Turnbull has simply dismisses all the post-election criticism as reckless. Yes, it’s reckless for people in the technology industry to comment on issues affecting the technology industry, according to Malcolm. It’s reckless for those who understand the technology, who work with the technology day in day out, who can offer insight into the issues — it’s reckless for them to offer an informed opinion. It’s reckless for people who work in the industry to comment on matters that will directly affect their livelihood.

Instead, in the bullshit alternate universe of politics inhabited by Malcolm and the Liberals, the best people to make technical decisions about essential communications infrastructure in Australia are economists and lawyers and accountants. Especially if they’re Liberal Party cronies.

The process of democracy will of course be best served by an endless — and no doubt expensive — series of committees and reports and reviews, overseen by technically clueless public servants, carefully selected to deliver the predetermined outcome Turnbull requires. Because that’s the democratic way, apparently. Many have accused Turnbull of hypocrisy, which seems kind of redundant, given hypocrisy is sadly not just synonymous with politics, but a prerequisite for success. Hypocrisy is the aioli in the gourmet bullshit burger of modern politics.

Hypocrisy is now as closely entwined in politics as sexting scandals. And as Australia’s 44th Parliament unfolds, I’m sure we’ll see plenty of both. Whether we want to or not. All that’s left is to wait and see exactly how we’re going to get screwed over by this new government of hypocritical, lying, cheating, backfliping and sexting parliamentary buffoons.

All we can do now is pray and hope that if there is going to be a sexting scandal, it doesn’t involve Clive Palmer. Because no one deserves to be subjected to that.

Given the level of crazy Clive displayed before he was elected, if he does become afflicted with whatever strange malady causes elected officials to think it’s a good idea firstly to take pictures of their genitals, and secondly to share those pictures with others via the internet, then let’s hope his aides can keep him in check. (“Clive, what are you doing? Clive, put that phone down and put your pants back on. Hey Clive, look over there! Clive! Put your pants on and look out the window there. Look! Over there. Isn’t that Wendi Deng doing secret Chinese spy shit? And over there! Look! Isn’t that a box of votes the AEC has been hiding from you? Look, Clive, look!“)

Let’s hope everyone can keep it in their pants during their term in office. After all, they have important work to do, screwing the geeks and the country out of a decent broadband network, amongst other things. And until us geeks learn how to play the bullshit game of politics, as distasteful as it may be, they’re going to keep screwing us. Until we work out how to play the game in the real world, no matter how much we blog or tweet or sign petitions, we’re not going to get the broadband we want or need — we’re going to get the broadband Turnbull and his corporate puppet masters want us to have.

Mandatory Internet Censorship to the Node

2013-09-06T00:00:00+10:00

So that was weird.

Yesterday, without fanfare, the Coalition released it’s “Policy to Enhance Online Safety for Children”. Because the the Liberal Party really, really cares about children. I’m sure it keeps them up at night. So much so they’re willing dedicate $10 million to keeping children safe online. Because if the government won’t keep children safe online then who will? Their parents? Or course not! Parenting is far too important a task to be left to parents — the government must be involved every step of the way! I’m sure this will be tax payer money well spent, and absolutely none of it will go towards hiring useless public servants or advisers who will just sit around all day downloading internet porn. I’m sure $10 million will buy lots of child safety online and none of that money will be wasted on pointless posturing, like some sort of mandatory internet censorship scheme. Oh, but wait, turns out that buried within the policy document was in fact a plan to introduce UK-style opt-out mandatory internet filtering. Coalition MPs Malcolm Turnbull and Paul Fletcher spent most of the afternoon promoting it.

Not surprisingly, controversy erupted. After all, this was the political party that had constantly criticised Labor’s eerily similar plans for compulsory internet censorship. And here we are, less than 2 days before an election, and without any debate, they spring a policy for mandatory internet censorship on us at the last minute. It seems just like the sort of underhanded thing the Liberals would do. Labor jumped all over it, shitting on the Liberal Party for having a policy of mandatory internet censorship, just like the Liberals have been shitting on Labor for their policy of mandatory internet censorship, and the circle of hypocrisy was complete.

But later on last night, just hours later, the Coalition backflipped. Turnbull released a statement saying the Coalition did not support mandatory internet filtering, and that the policy document was “poorly worded and incorrect.”

I suspect the policy document was “poorly worded” in that someone from the Liberal Party accidentally told the truth…in direct violation of Liberal Party policy. I assume the person responsible — or his scapegoat — was duly fired for this unexpected outbreak of honesty, and it won’t happen again. The last thing Australia needs is honesty in politics. Not when we’re doing so well with our current Merchants of Bullshit.

So the Liberals introduced mandatory internet censorship then took it back. They are truly the Jay Leno of IT policy.

It obviously raises many questions. Especially since Malcolm Turnbull was on the radio last night strongly arguing for the policy, before only hours later announcing the whole thing was a mistake. Not only that, he tweeted that he only read the policy shortly before going on air. On the one hand you’ve got to admire Malcolm Turnbull’s bullshitting skills for so convincingly standing up for a nonsensical policy he had only just read before going on air. He should give some pointers to Jaymes Diaz. On the other hand, it suggests Turnbull will happily weave a web of complete bullshit about any policy document put in front of him. So much for conviction politicians. The best we can expect these days are conviction bullshit artists. (Of which the Coaltion has many, though Jaymes Diaz is obviously not one of them.)

Then there’s the question of who the hell wrote the policy, if the ministers responsible for it are only reading it for the first time on the day it’s released to the public? The Liberals like to talk about the “faceless men” of the Labor party, but seems the Liberals have their own faceless men writing their policy documents — which no one even bothers to properly proof read before releasing.

Regardless of whether it was a genuine mistake or something more sinister, it doesn’t exactly inspire confidence in the Coalition.

Not that I really had much confidence in them to begin with. Certainly not when it comes to the world of IT. One of the things that has bothered me about the Liberal Party for a long time is the impression I get that they don’t really understand or even care about the IT industry. Or even any technology industry. Although to be fair, since in the past there haven’t been many votes to be had in this area, it’s something that’s been routinely ignored and neglected by both major parties.

Which made Labor’s National Broadband Network so unusually impressive. Here, for once, was a political party implementing genuinely visionary, nation-building technology infrastructure — and for the most part getting it right. Sure, it’s expensive and not perfect, but it’s pretty damn good on the technical merits — and long overdue, given the dire state of Telstra’s copper network. Surprisingly, the Labor Party more or less got it right as far as broadband goes. Do it once, and do it with fibre — to the premises. Although no doubt there was an element of self interest in this as well. Fundamentally, Labor’s NBN was borne of the realisation — way ahead of the Liberals — that the internet has become mainstream enough that there are now votes to be bought in this area.

But sadly not enough this time round. With a Coalition victory likely at tomorrow’s election, it’s a shame that Labor’s visionary plan to replace the crappy copper network with fibre and finally bring our broadband infrastructure into the 21st century, will be watered down with Turnbull’s technically inferior Fibre To The Node alternative.

That’s assuming we even get that far. The reality is once the Liberals get elected they will do whatever the hell they want, regardless of whatever “poorly worded” policy documents they release before the election. Including but not limited to finding whatever pretext they can to scrap the whole NBN, which they never wanted to build in the first place. Especially if they get control of the Senate. The only reason the Coalition even have a broadband policy is because Labor’s NBN plan proved to be so popular. However, I suspect many within the Liberal Party would love nothing more than to scrap the whole thing. Especially Tony Abbott.

The man who will likely be our next Prime Minister, is going into this election with two major and very expensive policies that he doesn’t believe in — the NBN and paid parental leave — and I suspect he’ll take any opportunity to scrap both. After all, this is the man who initially said if he was elected he’d rip Labor’s NBN out of the ground. This is the man who once said there would be compulsory paid maternity leave “Over this Government’s dead body“.

It therefore feels a bit ominous that just hours from an election, there’s a bizarre IT policy backflip, suggesting they’re making up policy as they go along. Which indicates — as I long suspected — that the Liberals don’t really care about the IT industry, and don’t really understand the IT industry. To them, it’s just another policy plaything. They will do whatever is most politically expedient, without regard for the best technical solution (hence Turnbull’s inferior FTTN NBN).

After the election, I wouldn’t be at all surprised if they find some excuse to scrap both the NBN and the paid parental leave. They might go for the John Howard approach — claim Labor left a big secret black hole in Australia’s finances, and therefore all those things they promised before the election are now unaffordable non-core promises, and therefore won’t happen. Goodbye NBN, and goodbye paid parental leave. They might not be able to get away with that — again — but I don’t think it’ll stop them trying.

It’s clear Tony Abbott thinks an NBN is a waste of money, but seems he thinks nothing of throwing millions of dollars of tax payer funds at a chocolate company. After criticizing Kevin Rudd for giving government assistance to the struggling car manufacturing industry, Abbott has promised $16 million to a profitable multi-national corporation to upgrade a chocolate factory in Tasmania. Just the thing we need to be spending money on in the midst of an obesity crisis and diabetes epidemic — chocolate! Forget about building the broadband infrastructure of the future — chocolate manufacturing is what’s most important! So we might not get decent broadband after the election, but we can always gorge ourselves into a diabetic coma on Australian-made, government-subsidized chocolate to try and forget about how our ADSL connection keeps dropping out.

The Australian Broadband Hustle

2013-09-01T00:00:00+10:00

It got to the point I just had no choice. Had to do it. I’d had enough.

I’d had enough of the increasingly frequent ADSL dropouts on my increasingly less reliable TPG broadband service.

I’d had enough of a Telstra phone line with so much noise, static and crackling sounds that it was almost impossible to hear the person on the other end.

I’d had enough of both my phone and internet going to hell every time it rained.

I’d had enough of an ADSL line so unreliable it would drop out every few minutes.

I’d had enough of Telstra’s copper network in my area which, after years of intermittent problems, had obviously degraded to the point of being completely unusable.

I’d had enough of the frequent but futile conversations with TPG’s support drones, who would hear the noise on the line and insist it was Telstra’s fault the ADSL was constantly dropping out. After all, a line so noisy was bound to cause ADSL interference, they insisted.

I’d had enough of repeatedly reporting the fault to Telstra, only to have a technician arrive few days later and tell me there wasn’t anything wrong with the line, and it was TPG’s fault the ADSL was constantly dropping out. Or my phone’s fault. Or the line filter’s fault. Or my modem’s fault. Or the cables’ fault. So I replaced the modem, the phones, the line filter, and all the cables. Still, the same problems persisted. So then it was suddenly my home’s fault for being so damn distant from the exchange. Or my star sign’s fault. Or the fault of those pesky cosmic rays.

I’d had enough of constantly going around in circles with Telstra and TPG, each one blaming the other for the dropouts on my ADSL service and noise on my phone line.

I’d had enough of both TPG and Telstra being all too happy to take my money, while neither was willing to accept any accountability for problems with the sub-standard services they were providing.

I’d had enough of paying for a phone and internet services that had both become completely useless.

So I had no choice but to switch from ADSL2+ to cable internet. Fortunately, I am in an area where cable internet is available. Unfortunately, it’s more expensive. Fortunately, this was an opportunity to get rid of Telstra completely, as I had the option of another provider.

‘Yes’, that other provider.

So it was goodbye and good riddance to Telstra. Useless fuckers. They treat their customers with contempt in a way only a government sanctioned monopoly can.

I switched to cable internet, and for the first time in years I experienced a stable, reliable, and fast internet connection. And a phone line clear as a bell. Miraculously enough, the same cables and phone equipment that the Telstra technicians blamed for my ADSL woes are working fine plugged into the cable modem. How about that? Is it possible the Telstra technicians were bullshitting me?

‘Yes’, I believe it is.

It’s amazing what a difference decent technology makes to the user experience. This is a point that seems to have been lost in the NBN debate, which has become mostly focused on the differences in speed between Labor’s NBN plan and the Coalition’s crappy alternative. But the choice of technology is crucial to other factors — like reliability — that makes a big difference between a good and bad internet experience. No matter how fast (or slow) a broadband connection is, it’s going to be a frustrating experience if it drops out at crucial times — which is likely to happen with the Coalition’s FTTN network as it uses the existing copper network and is therefore prone to the sort of problems that the copper network is renown for.

These problems went away when I went away from the copper network and went to cable. This resolved the technical problems. But did this other internet provider manage to cause other problems?

‘Yes’, of course they did.

All I wanted was reliable internet and phone service. (Is that really too much to ask? In Australia, seems the answer is ‘Yes’.) But of course I had no choice but to select a “bundle”. Despite checking every “opt out” check box I could find when signing up, I still got a whole bunch of crap I didn’t want. Fortunately, they were running a promotion at the time I signed up where (almost) all the crap I didn’t want was “free”. Kind of. Once I got around to setting up the “free” PVR included in the bundle, it wasn’t hard to see why they were giving it away for “free” — it’s a device designed mainly to entice users to select any number of additional services that will be automatically added to the monthly bill.

Those extra charges are likely to go unnoticed, because the bills are a deliberately complicated work of byzantine obfuscation.

This internet provider (‘Yes’, that one) easily wins the prize for most complicated and difficult to decipher bills I have ever encountered. I suspect this is all intentional — an attempt to confuse customers as much as possible, so that they can scam as much money out of customers as possible. Although I do find it somewhat amusing that this company in question (‘Yes’, that one) has a section on their website devoted to explaining their bills. Doesn’t this represent a failure of user experience with regards to their billing?

‘Yes,’ I believe it does.

If they actually cared about their customers understanding their bills, couldn’t they actually just simplify their bills?

‘Yes,’ of course they could.

But they don’t particularly want to , because then customers would be more likely to notice any billing errors.

Like I did, after a few months, when I received a bill for $125. Since I was on an $85 plan, this didn’t seem quite right.

Is $125 more than $85?

‘Yes’

Far be it from me to question the wisdom of large corporations. If there’s one thing I learnt from my Catholic school education, it’s that no good can come from thinking for yourself and questioning authority. But because I don’t trust my Catholic school education, I turned to Excel. Because I totally trust Microsoft.

So tell me Excel, is an Australian telecommunications company trying to rip me off?

‘Yes.’ Indeed they are.

Imagine my complete lack of surprise at this point. So this is the choice I have in the marketplace as a consumer of internet services. I can either pay for an ADSL service that just doesn’t work, or I can pay a premium for a functional cable internet, including extras I don’t want, and get overcharged at every turn. Free market competition at its best, I’m sure.

I called up their helpline and spoke to someone from the Land of the Distant Call Centre, who might have said her name was “Mary”, although I somehow suspect it wasn’t.

Turns out I was being charged for the supposedly “free” crap I didn’t want in the first place, backdated to the time I switched over to this new cable internet service.

“Mary” (if that was her real name) agreed to credit back the amount overcharged. So all good right? No harm, no foul? Maybe. At this point, after my experiences with Telstra, I have no goodwill left for telecommunications companies. At this point, I choose to take the most cynical and sinister interpretation of events. At this point, I choose to believe the overcharging was intentional. It’s all part of standard operating procedures for telecommunications providers in Australia — create deliberately over-complicated plans/bundles, with lots of extras customers probably don’t want, confuse/distract the customer with ridiculously complex bills, then deliberately overcharge them and hope they don’t notice. Worst case scenario, the customer calls and complains, and maybe gets a refund. But for each time this happens, there are probably far more cases where they customer doesn’t realise they’ve been ripped off, which means these corporate con artists get to keep their ill-gotten gains.

Although calling them con-artists is probably an insult to accomplished con-artists everywhere. It would imply that there is some “artistry”, or craft, or at least competence, to what the internet companies are doing. But there’s not even that. The telecommunications industry in Australia is less an elaborate confidence trick, and more a simple forthright fraud. It’s like a street shell game where there is no ball, no prize, and the moving cups are just a momentary distraction before beating the unsuspecting mark to the ground, stealing their wallet and phone, then calmly standing there as if nothing happened, knowing there will be no repercussions. It’s less con artist and more highway robbery.

I’m not sure what’s worse. The fact all Australian telecommunications companies routinely fuck their customers over any chance they get? Or that they fuck their customers in such an obvious and artless way? It’s as if they’re not even trying! If I’m going to be fucked over by an uncaring, unethical, unscrupulous corporation, I’d like them to at least put some effort into it. If I’m going to be fucked over, I’d rather be fucked good! Saying something will be included in the price then charging extra anyway and hoping the customer won’t notice? How unoriginal! Put some imagination into it! Do it with flair!

Of course, screwing their customers is hardly confined to the telecommunications industry. (I should at least mention Australian banks, if only in passing.) But what’s most disturbing is just how common this kind of corporate laziness and contempt for customers and has become. This culture of mediocrity in Australian business has gotten to the point of making a mockery of the whole concept of running a private sector business. In recent months, we’ve had a car company extort the government, threatening to stop manufacturing cars in Australia unless it gets more government handouts; we’ve had an electricity company demand the government pay them to not produce electricity.

And here I was naively thinking the idea of a business is to provide an actual product or service in exchange for money from customers. I guess I should stick to software development, as I’m obviously missing something about modern business practises.

Although maybe I’m being a bit too harsh. After all, if the government is stupid enough to give out free money to corporations who don’t deserve it, why shouldn’t those corporations do their best to get as much cash out of the tax payer as possible? Surely it must be hard to constantly compete and innovate and carve out a competitive advantage in a free market? So why bother if you don’t have to? Why should Australian car companies adjust to changing market conditions and consumer preferences, when they can just continue making the same old crappy cars no one want to buy anymore, safe in the knowledge the government will continue paying them to do so, even if consumers won’t? Why should electricity companies produce electricity if they can get idiot politicians to pay them not to produce electricity? It’s much easier that way. And more environmentally friendly too. Just think of the reduced carbon footprint? And what do we need electricity for anyway?

And why should Telstra fix the copper network near my place when they’ve got a monopoly on the network? Why wouldn’t Telstra’s competitors deliberately overcharge me, just to see if they can get away with it? They no doubt realise that if I’m a customer of theirs, it’s because I’m out of other options, so who else am I going to churn to? Telstra? Ha! And it’s not as if the regulators are going to do anything. So the telecommunications companies can do whatever the hell they want. (How much to charge for international roaming? As much as they fucking well can!)

So it’s against this backdrop of corporate laziness, and culture of mediocrity, we have the Labor’s National Broadband Network.

Tony Abbott’s initial reaction to Labor’s FTTP NBN proposal was to say if he was elected, Labor’s NBN would be ripped out of the ground. He angrily insisted broadband infrastructure should be build by the private sector. Really Tony? The private sector? The same private sector unable to provide me with a clear analogue phone line? The same private sector unable to provide me with a reliable ADSL service? The same private sector unable to maintain the current copper infrastructure? The same private sector that finds the prospect of billing me the same $85 amount every month too difficult? The same private sector that can’t provide me with reliable mobile internet? (I’m looking at you Vodafone.) The same private sector that routines charges customers for a service that doesn’t work? (I’m looking at you every single ADSL provider.) The same private sector that’s concerned only with fucking over their customers?

For starters, there’s the obvious question of why the private sector hasn’t built a decent broadband network by now? Why are we still stuck with this ancient, crumbling copper network for our communication infrastructure? Abbott can’t blame Labor for this. The Liberals were in power for 11 years under John Howard’s leadership, during a period of great technological growth and innovation in other parts of the world. But for their part, it seemed the Howard government thought Australia’s role in the great internet boom was comic relief, appointing a series of clueless and incompetent IT ministers, and making us an international laughing stock in technology circles. Worse, the Howard government privatised Telstra without structural separation, so even privatised it still remains essentially a government sanctioned monopoly. This not only prevents proper competition in the sector, but also goes a long way to preventing proper investment, making it difficult, or even impossible, for a private company to get a reasonable return on any investment in broadband infrastructure.

Saying the private sector should build a next generation broadband network is all well and good, but if it hasn’t happened by now it never will. Which makes Tony Abbott and the Liberal’s position on broadband so frustratingly naive, short-sighted and stupid. They’re meant to be the political party that most represents the interests of business, yet they don’t seem to understand the current culture of mediocrity that pervades the Australian business landscape. Australian business leaders seem determined to prove themselves incapable of even wiping their asses without government assistance. Yet Tony Abbott expects any of them to have the forward-thinking vision, initiative and ability to build Australia’s next generation broadband infrastructure? What kind of dreamworld is he living in? Telstra can’t even maintain the current infrastructure to a working standard!

It also represents a deep misunderstanding of the Australian telecommunications industry — in that we don’t really have one. Australian telecommunications companies — from Telstra and the various ISPs to the mobile phone providers and pretty much everyone in between — aren’t in the business of providing phone and internet services. That’s just a side-effect. They’re actually in the business of fucking over their customers in the most direct ways possible. It all a scam. A confidence trick. A hustle. And not a particularly clever one at that. The fact any “telecommunications” are happening at all is probably just because a few competent people accidentally got hired once. Or someone didn’t get a memo. The last thing any of these companies want to do is build communications infrastructure. (Just look at Vodafone’s 3G network!) In fact, they would rather not, if they can help it. Building infrastructure is a unwelcome distraction from their core business model — screwing their customers. (And sometimes screwing the Tax Payer.)

Labor’s NBN plan, for it’s faults, at least takes all this into account. Labor seems to get it. They understand the private sector will never build a decent broadband network. They understand the copper network is well past its use-by date, and needs to be replaced by something better (i.e. fibre). They understand that for better or worse, the internet is becoming increasingly ingrained into the fabric of society and business, so it’s vital to our future economic prosperity that fast and reliable internet services are available. They understand the problem of Telstra as the monopolistic elephant in the room, which therefore requires it to be structurally separated for the good of the industry and the country — even if it risks replacing one obnoxious government sanctioned monopoly with another.

Malcolm Turnbull might have managed to do the seemingly impossible and softened Abbott’s hardline stance on the NBN (although losing an election partly because of a misguided and unpopular broadband policy stance probably also helped), with the Coalition now grudgingly promising to build some kind of half-assed broadband network. I suspect Turnbull sat down with Abbott and explained to him using small words and visual aids that Australia’s telecommunications industry was actually one great big fraud perpetrated on the Australian people, and the natives were beginning to catch on and get restless. Since Turnbull has experience as a lawyer, banker and ISP chairman, he no doubt knows all about defrauding the public. A little too well, perhaps.

The problem with the Coalitions NBN policy is that it shows no understanding of the realities of the current state of play that Labor’s does. Labor’s NBN acknowledges that the current scammy system has become obsolete and therefore big (and expensive) changes are needed. Malcolm’s NBN is predicated on a convenient fiction that the existing copper network is still good enough — which it is not. Labor’s NBN is an acknowledgement that broadband in Australia has become a scam and need to be replaced (albeit with possibly grander scam, but at least one that revolves around working, modern technology). Malcolm’s NBN is just a continuation of the current scam. Malcolm’s NBN will be just another chapter in the great Australian broadband hustle — where corporations get rich, but consumers get screwed. The same scam, except with big green node cabinets on every street corner.

With the Coalition’s NBN, it’s likely to be the same situation we currently have, with Telstra as a government sanctioned monopoly without corporate accountability, and no particularly good choices in the marketplace for consumers. We either get a crappy service that barely works, or pay exorbitant prices to get anything even half decent — and no proper competition in the market to keep the corporations honest, leaving them free to continue their scammy ways. Which means the consumer of broadband services in Australia is much like a pornstarlet in a gangbang scene — no matter who they turn to, they’re going to get perfunctorily fucked. One way or another. Because that’s the whole idea, isn’t it Malcolm?

‘Yes’

Extract emails from Gmail with Python via IMAP

2013-07-25T21:00:00+10:00

I have a filter setup in a Gmail account to automatically collect what are essentially automatically generated emails from a particular source, and file them neatly away into a label, leaving the inbox relatively uncluttered with their consistently formatted regularness. Thing is, those particular emails are not achieving their potential for maximum usefulness to me by just sitting there in my Gmail account. And with their predictable content, they are ripe candidates for parsing and extracting the useful data, and storing it in more useful form, like a database or spreadsheet. But instead they just pile up, and silently mock my supposed geek cred with each passing day. Well I have had enough of their judgmental presence! It is high time I finally worked out a way to programmatically download and parse their sorry arse. And anyway, if the NSA is allegedly algorithmically accessing my emails why shouldn’t I? So as part of my continuing effort to Script My Life, I once again turn to trusty Python to make my life easier — one hackily bespoke script at a time.

The nice thing about Python is that there’s a module for just about everything. The not-so-nice thing about Python is that there’s usually more than one module for everything. And often a bewildering number of packages offering the same functionality. Usually all deprecated — except for one. Or possibly all “un-Pythonic” — except for one.

So while there may be more than one way to programmatically get emails out of Gmail with python, one option is IMAP. Gmail can be accessed via IMAP, and conveniently enough the Python Standard Library has an IMAP interface, so it appears not unreasonable to use IMAP. Unless you’re reading this in a future where IMAP has been deprecated. Or deemed un-Pythonic.

First step is to create an IMAP4 instance, preferably the SSL variant for security, connected to the Gmail server at imap.gmail.com:

#!/usr/bin/env python

import sys
import imaplib
import getpass
import email
import datetime

M = imaplib.IMAP4_SSL('imap.gmail.com')

Next we can attempt to login. If the login fails, an exception of type imaplib.IMAP4.error: will be raised:

try:
    M.login('notatallawhistleblowerIswear@gmail.com', getpass.getpass())
except imaplib.IMAP4.error:
    print "LOGIN FAILED!!! "
    # ... exit or deal with failure...

If the login is successful, we can now do IMAPy things with our IMAP4 object. Most methods of IMAP4 return a tuple where the first element is the return status of the operation (usually 'OK' for success), and the second element will be either a string or tuple with data from the operation.

For example, to get a list of mailboxes on the server, we can call list():

rv, mailboxes = M.list()
if rv == 'OK':
    print "Mailboxes:"
    print mailboxes

With Gmail, this will return a list of labels. To open one of the mailboxes/labels, call select():

rv, data = M.select("Top Secret/PRISM Documents")
if rv == 'OK':
    print "Processing mailbox...\n"
    process_mailbox(M) # ... do something with emails, see below ...
    M.close()
M.logout()

So with the mailbox selected, we can now get the emails within it. For example, we can get all the emails in the selected mailbox and for each one output the message number, subject, and date:

# Note: This function definition needs to be placed
#       before the previous block of code that calls it.
def process_mailbox(M):
  rv, data = M.search(None, "ALL")
  if rv != 'OK':
      print "No messages found!"
      return

  for num in data[0].split():
      rv, data = M.fetch(num, '(RFC822)')
      if rv != 'OK':
          print "ERROR getting message", num
          return

      msg = email.message_from_string(data[0][1])
      print 'Message %s: %s' % (num, msg['Subject'])
      print 'Raw Date:', msg['Date']
      date_tuple = email.utils.parsedate_tz(msg['Date'])
      if date_tuple:
          local_date = datetime.datetime.fromtimestamp(
              email.utils.mktime_tz(date_tuple))
          print "Local Date:", \
              local_date.strftime("%a, %d %b %Y %H:%M:%S")

We use the search() method to get a list of message sequence numbers, then loop over these, calling fetch() to get the actual messages.

fetch() returns the raw message contents. To avoid having to parse the actual message data from fetch() ourselves, we can use the email package from the standard library. Once again, there are a few different packages floating around for doing this kind of thing, but I think email is currently the one least likely to get you down-voted on Stack Overflow.

message_from_string() returns a message object, and we can then access header items as a dictionary on that object.

Which brings us to the “Date” header, and the potentially thorny issue of date and timezone. If you don’t care about the date/time the emails were sent, then things are much simpler. But if you do care about such matters, note that the contents of the “Date” header may vary depending on the email client sending the email, and the timezone of the sender. Reliably converting to local time can be surprisingly tricky. Python is once again comes to the rescue with numerous modules promising to assist you with all your deepest darkest date conversion needs. The code snippet above shows one possible way of converting to local time, using the capabilities of email.util. It’s currently working well enough for my purposes, but there may be better ways to accomplish this.

The message body can be obtained by calling msg.get_payload(), which will return the payload data as a string (if the message is not multi-part). For text messages, you could then parse the data using regular expressions. For parsing contents of HTML emails however, you must not use regular expressions. Ever. Or you will feel the Lovecraftian wrath of the Stack Overflow minions. Instead, use a HTML parser, or higher level scraper like Beautiful Soup.

UPDATE:

A complete version of the above code is available in this gist.
For a variation on the theme, see also this gist: a script that simply dumps all emails in an IMAP folder to files.

Subclassing QTableWidget into submission

2013-06-21T10:19:00+10:00

Python example of subclassing a Qt QTableWidget control and overriding copy behavior so that the entire selected row is copied to the clipboard, and stored in a Excel-friendly format for proper pasting.

Hello there QTableWidget, you look like just the sort of Qt control I’m looking for. I want to stick some data into you. But once the data is inside you, I might want to do other things. Like perhaps copy and paste rows of data into Excel or some other spreadsheet. I wouldn’t have thought that’s an overly unusual request. But looks like you won’t do that without some overt persuasion. Seems like you’re not into data swapping with other applications. Well then, you’ve left me no choice but to subclass your ass:

#!/usr/bin/env python

import sys
from PySide.QtCore import *
from PySide.QtGui import *

class TableWidgetCustom(QTableWidget):
    def __init__(self, parent=None):
        super(TableWidgetCustom, self).__init__(parent)

    def keyPressEvent(self, event):
        if event.matches(QKeySequence.Copy):
            self.copy()
        else:
            QTableWidget.keyPressEvent(self, event)

    def copy(self):
        selection = self.selectionModel()
        indexes = selection.selectedRows()
        if len(indexes) < 1:
            # No row selected
            return
        text = ''
        for idx in indexes:
            row = idx.row()
            for col in range(0, self.columnCount()):
                item = self.item(row, col)
                if item:
                    text += item.text()
                text += '\t'
            text += '\n'
        QApplication.clipboard().setText(text);

See QTableWidget, that wasn’t so bad after all now, was it?

I expected the Qt table widgets would implement proper copy and paste of rows by default, especially if options like QTableWidget.SelectRows are set. But seems I was being overly optimistic. No matter what options I tried, it would only copy the text in the single cell that was clicked on, regardless of how many cells were actually highlighted. So had to resort to subclassing. The above is a fairly minimal implementation that seems to work well enough for my purposes.

For data being pasted from the clipboard, apparently Excel expects columns to be tab delimited, with rows ending with a newline. The copy() method above attempts to satisfy Excel’s desires in this regard, and places the copied text in the required format onto the clipboard.

To see this in action, the customised QTableWidget can be used as normal:

class Form(QDialog):

    def __init__(self, parent=None):
        super(Form, self).__init__(parent)
        tableLabel = QLabel("Behold, some data, in a table:")
        self.customTable = TableWidgetCustom()
        self.customTable.setEditTriggers(QTableWidget.NoEditTriggers)
        self.customTable.setSelectionBehavior(QTableWidget.SelectRows)
        self.customTable.setSelectionMode(
                             QTableWidget.SingleSelection)
        layout = QVBoxLayout()
        layout.addWidget(tableLabel)
        layout.addWidget(self.customTable)
        self.setLayout(layout)
        self.setWindowTitle("QTableWidget Copy and Paste Example")
        self.customTable.setColumnCount(5)
        self.customTable.setRowCount(3)
        for row in range(3):
            for col in range(5):
                item = QTableWidgetItem("Row %d Column %d" % (row+1,
                                                             col+1))
                self.customTable.setItem(row, col, item)

app = QApplication(sys.argv)
form = Form()
form.show()
app.exec_()

The above code creates a dialog box with an instance of our customised QTableWidget and populates it with some dummy data. Selecting a row and pressing Control-C will copy that row to the clipboard, and it can then be pasted into Excel (also works in LibreOffice). But if you were to replace the TableWidgetCustom control with a standard QTableWidget, you’d see that Control-C will only copy the data in one cell, not the entire row.

HTML meta mischief in Pelican blog posts

2013-05-25T18:45:00+10:00

For some reason recently I got to thinking about HTML meta elements, and thinking about these kinds of things often leads to acts of anal retentive geekiness. Like noticing that few Pelican themes do much with HTML meta elements, and usually nothing with the description tag in particular. And it was the HTML meta description tag I was thinking about most.

I was thinking about whether it’s even worth thinking about, for it seems these days HTML meta elements are considered somewhat unimportant as far as SEO goes. Search engines may or may not use a meta description tag (or part thereof) as summary snippets on search results pages, which may influence click through rates, for those that care about such things.

I’m not sure I do, but it did get me thinking a bit about the way Pelican handles article summaries and other such things.

For some posts I prefer to provide my own summary, while for others I’m happy for Pelican to extract a certain number of words from the start.

In the same way, for some posts, I might be happy for search engines to come up with their own summaries, which are likely to be the first sentence or two. However, there are definitely some posts where not only would I like to have some say in the summary that’s displayed on a search engine result page, but I would also be willing to go to great lengths to make sure it isn’t anything from the first paragraph. Especially if I’ve decided to be deliberately long-winded or scatological in the first paragraph. (Of course, some might suggest the best course of action is not to be long-winded or scatological, but what would be the fun of that?)

So I decided I did want to have more control over these kinds of things, and therefore having the ability to optionally specify a meta description tag for blog posts would be a useful feature to have in my increasingly hacky custom theme.

A possibly viable option for some might be to just to take the article summary and use that as the content of the meta description too. But I wanted more control than that — more often than not I would want the summary to be different to that of the meta description. So I took a different approach.

With Pelican it’s possible to add arbitrary metadata tags to a blog post, and the value of this metadata tag will be available in templates as an attribute of the article object. This is great, because it means I can add all kinds of metadata tags that make sense to me (if no one else) and modify my theme’s templates to render that metadata as appropriate.

For example, let’s say we have the following blog post in reStructuredText format:

Mild Self Loathing
##################

:date: 2010-10-03 10:20
:tags: mindless drivel, incoherent whining
:category: self indulgent rambling
:slug: mild-self-loathing
:author: me
:description: This text will be put in the meta description tag.

Main text of the blog entry.

Note it contains an arbitrary metadata tag called “description”.

In the theme templates, the value of this description tag will be available as article.description.

So in the theme’s base.html file we could add the following:

<meta name="description" content="{% block metadesc %}{% endblock %}" />

This would define an empty meta description by default. (Alternatively, we could have instead specified some default text for the description tag.)

We can then override this block in the theme’s article.html to define a meta description specific to the blog post:

{% block metadesc %}{{ article.description|e }}{% endblock %}

The value of article.description will be the value given in the blog post’s metadata. So in the above example, that blog post will include the following HTML meta element in the generated HTML:

<meta name="description" content="This text will be put in the meta description tag." />

The same thing can be done with Pelican pages, except in this case the object we’re working with in the template will be called “page”. So a similar block in the theme’s page.html file might look something like:

{% block metadesc %}
{% if page.description %}{{ page.description|e }}
{% else%}A page in my blog called {{ page.title }}
{% endif %}
{% endblock %}

The above will use a description metatag if one is specified, otherwise it will create a generic one using the page title.

Of course, there are other more esoteric things that can be done with HTML meta elements. For instance, if you’d prefer tag pages not clutter up potential search engine results, you can suggest that search engines not index them by adding a robots meta tag to the tag archive page. In the theme’s base.html add this line:

{% block meta_other %}{% endblock %}

Then in the theme’s tag.html file, override the meta_other block like this:

{% block meta_other %}<meta name="robots" content="noindex" />
{% endblock %}

Now, the tag archive pages will have a robots noindex directive in the head.

We can of course conditionally include or exclude content depending on the value of metadata. So, for example, we could do something like this in article.html to prevent draft posts from being indexed by search engines:

{% block meta_other %}
{% if article.status == 'draft' %}
<meta name="robots" content="noindex" />
{% else %}{% endif %}
{% endblock %}

The above example used the standard “status” metadata, but of course, given we can add all the arbitrary metadata we like, similar things can be done with our own custom metadata tags, which makes it relatively straightforward to add custom conditional features to Pelican themes.

I’ll write a script instead…

2013-04-12T14:32:00+10:00

A Python script to convert date fields in a CSV file from MM/DD/YYYY to DD/MM/YYYY format — because Excel wouldn’t do it properly!

I imagine there must be many jobs, especially in and around the finance industry, where using Excel is part of daily life. I feel sorry for those people.

I find myself using spreadsheets quite a bit lately, and it’s often a frustrating experience. There’s a lot of annoyingly repetitive pointing and clicking for one thing, which is frustrating for a Unixy guy like myself. Then there’s Excel’s remarkable ability to find unique and unexpected ways to mangle the data I’m manipulating each step of the way. (And from what I’ve seen, Libre Office isn’t much better.) But what really gets to me is when I have to do something that seems like it should be relatively simple to do, but turns out to be anything but simple.

For example, I recently found myself wanting to import a CSV file into Excel. A seemingly straightforward enough task in itself, assuming the CSV file is properly formatted, which in this case it was. The problem was that the CSV file in question had a date-time field, and the date part was in US format (MM/DD/YYYY hh:mm), and I wanted it in “Australian” format. (DD/MM/YYY hh:mm). Well, to be exact, I didn’t’ really care about the date format, but I wanted it converted into a proper “date” field in Excel with the correct value, so I could sort by date. This proved to be a problem. Annoyingly though, the functionality appears to be there. There is a “Text Import Wizard” which in theory allows date fields to be marked, and even the order (e.g. “D/M/Y”) to be specified. It just didn’t work. For me. For this file. I think it was getting confused by the time part, and despite much trial and error and fiddling on my part, it would not parse the date-time field correctly.

There might have been a way to get this work. And I’m pretty sure I did get something like this working in the past (maybe using Libre Office??). But in frustration, I thought, “screw Excel, I’ll write a script to do this instead.”

And so I did. I wrote a Python script to reformat date-time fields in a CSV file just so I could then load the CSV file into Excel and have it properly recognise the date-time values. What fun. There’s time well spent.

So for anyone who might be even remotely interested — and so that they don’t have to do this themselves — here is the script.

It takes 3 command line arguments:

Input filename
Output filename
Index of the date-time field to reformat (zero-based)

It makes the following assumptions:

File is comma delimited.
First line in the file is a header, and discarded, so output is without a header line.
In date-time fields, only the date part is modified. If there’s any text after the date part, it’s included in the results unmodified, so both date and date-time fields should be handled properly.

#!/usr/bin/env python

import sys
import re
import csv

if (len(sys.argv) > 3):
    input_filename = sys.argv[1]
    print "Using input file:", input_filename
    output_filename = sys.argv[2]
    print "Using output file:", output_filename
    field_index = int(sys.argv[3])
else:
    sys.exit("Usage: csv_us2au_date.py <inputfile> <outputfile> <fieldindex>")

prog = re.compile(r'(\d+)/(\d+)/(\d+)(.*)')

of = open(output_filename, "wb")

with open(input_filename, 'rb') as csvfile:
    writer = csv.writer(of)
    reader = csv.reader(csvfile)
    next(csvfile)       # skip first line!

    for row in reader:
        m = prog.search(row[field_index])
        if m is None:
            print "INVALID DATE AT LINE %d" % (reader.line_num)
            v = row[field_index]
            row[field_index] = '"' + v + '"'
        else:
            row[field_index] = "%02d/%02d/%d" % (int(m.group(2)),
                                                 int(m.group(1)),
                                                 int(m.group(3)))
            if m.group(4):
                row[field_index] += m.group(4)
        writer.writerow(row)

So nothing earth-shattering there, just annoying this is what I had to resort to in an attempt to get Excel to do what I wanted it to.

Which got me thinking about how, as a coder, there is a tendency to re-invent the wheel (“That way of doing this is fucked, I want to do things my way, because my way will be better!”) This is something I (and I imagine most coders) have to be careful of, and often fight against.

For one thing, it’s not always efficient in terms of time and energy. In this case, it’s debatable whether I saved any time. It’s not a long or complex script by any means, but the input file wasn’t particularly big either. It probably would have taken me a similar amount of time just to manually adjust the date fields. Although that would have been much less “fun”.

(The problem was, I didn’t stop just there. Once I started coding, I kept coding! I ended up writing a whole bunch of far less trivial scripts to essentially automate all the data processing I was intending to do in Excel manually. This definitely took more time than it would have taken to do manually, but if nothing else it was an interesting exercise in Python programming and using SQLAlchemy. It also means doing the same thing when Tax Return time comes round again next year should be a lot simpler — assuming the broker doesn’t change the format of it’s CSV export file!)

On the other hand, many a great software product or open source project has started just this way — with someone cracking the shits and turning to coding to solve a particular problem or automate some tedious task.

This is all well and good for those of us who can code — for those of us who do have the option of saying, “screw it, I’ll write a script to do this instead”. But what about those who don’t have that “luxury”? What about those who’s only interest in computers and software is as a tool to get things done? Like those people who might have to use spreadsheets as part of their day job (for want of a more specialised tool)? What do they do when face with some simple text processing task, like in this case converting a date-time format. What then? Write a VBA macro? Sure, that’s presumably why there’s a scripting language is embedded into Excel, to allow (power) users to deal with (advanced/unusual) situations the software by default cannot handle (easily). But that means learning VBA, and by extension, learning something about scripting/programming. Not everyone is interested or cut out to be a coder, so this isn’t going to be a great option for them. Which leaves lots of manual pointing and clicking.

Of course, I’m looking at this from the point of view of a programmer. I know that converting between date formats is a simple text processing task, the sort of simple text processing task computers are well suited for. Which is why I am frustrated to find I cannot get Excel to do it easily. Perhaps someone less computer-literate might just resign themselves to do it manually and not give it a second thought.

There are some that therefore say this is why some coding knowledge is useful even for non-coders in this day and age. Well, kind of. If someone is interested in learning to code, I’m all for it, have at it. But I think the reality is not everyone is cut out to be a (good) coder. And there are some people who just have no interest in coding, which I think is fine. Computers after all are just a tool, a machine, a labour saving device. I don’t think computer users should need to be coders to use computers effectively, in the same way car drivers shouldn’t need to be mechanics to drive a car! If anything, I think the onus is on us coders to do a better job of creating and designing software that allows users to do what they need to do as efficiently as possible and with the minimum amount of annoyances possible.

In this respect, it seems despite the advances that have been made in technology and human-computer interaction, we still have a long way to go to make the technology stay out of the way for users who just want to get things done, without having to tinker under the hood. Well, at least that’s the case with Excel.

Populate a Qt combo box from a database table in PyQt

2013-02-05T14:46:00+11:00

Qt provides some useful mappers for easily linking relational database tables with table/grid widgets. This works well for quickly producing simple CRUD applications. But sometimes we might want to escape the limitations of table and grid controls, and do something fancier — something to provide users with a more customised, fine-grained interface, which usually ends up being something based on forms/dialogs. So we design a form, populate it with controls, and we are then faced with having to populate individual controls from database tables.

I did this recently, and was then faced by what I assume would be a fairly common situation: I had a simple database table for storing status values, and wanted to populate a combo box on a dialog with data from that table (and retrieve the matching ID the user has selected at some point).

Surprisingly I couldn’t find a simple example of this online or in the Summerfield book, so I thought I’d write about the approach I used. (Let me know in the comments if there’s a better way!)

For example, in a hypothetical contact management application, we might have an SQLite table for storing the “category” of phone numbers:

CREATE TABLE phone_number_type (
     id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
     label TEXT NOT NULL
);

So in this case we essentially have a glorified enum (but perhaps one which the user can add/modify, so it can’t just be hard-coded into the application). At a minimum, there would be an integer primary key field, along with a corresponding string for the category name. (There could possibly be other fields as well, but we can happily ignore them for the task at hand.)

We can populate the table with some data like so:

INSERT INTO "phone_number_type" VALUES(1,'Home');
INSERT INTO "phone_number_type" VALUES(2,'Work');
INSERT INTO "phone_number_type" VALUES(3,'Mobile');
INSERT INTO "phone_number_type" VALUES(4,'Mistress');
INSERT INTO "phone_number_type" VALUES(5,'Meth Dealer');

Since in such cases we’re usually dealing with a small amount of data, a valid approach might be to just load all the data from the table into a data structure in memory, and populate the combo box from that as appropriate. However, this approach is less than ideal when the table in question contains data that the user can modify, and therefore might change frequently. It also means writing repetitive boilerplate code.

In this case we can use the Qt Framework’s QSqlTableModel class to help us out. The following example creates a simple dialog box for editing phone numbers. The dialog box will contain a combo box widget populated with data from the phone_number_type table, as defined above.

#!/usr/bin/env python
import os
import sys
from PyQt4.QtCore import *
from PyQt4.QtGui import *
from PyQt4.QtSql import *

CONFIG_DATABASE_PATH = "./"
CONFIG_DATABASE_NAME = "comboboxexample.db"

class EditItemDlg(QDialog):
    def __init__(self, parent=None):
        super(EditItemDlg, self).__init__(parent)
        self.setWindowTitle("Edit Phone Number")

        # Create model
        self.model = QSqlTableModel(self)
        self.model.setTable("phone_number_type")
        self.model.select()

        buttonBox = QDialogButtonBox(QDialogButtonBox.Ok |
                                     QDialogButtonBox.Cancel)
        buttonBox.button(QDialogButtonBox.Ok).setDefault(True)

        self.phoneEdit = QLineEdit()
        phoneLabel = QLabel("&Phone Number:")
        phoneLabel.setBuddy(self.phoneEdit)

        # Create combo and set its model
        self.typeComboBox = QComboBox()
        self.typeComboBox.setModel(self.model)
        self.typeComboBox.setModelColumn(
                          self.model.fieldIndex("label"))
        typeLabel = QLabel("&Type:")
        typeLabel.setBuddy(self.typeComboBox)

        controlLayout = QGridLayout()
        controlLayout.addWidget(phoneLabel, 0, 0)
        controlLayout.addWidget(self.phoneEdit, 0, 1)
        controlLayout.addWidget(typeLabel, 0, 2)
        controlLayout.addWidget(self.typeComboBox, 0, 3)
        controlLayout.addWidget(buttonBox, 1, 0, 1, 4,
                                Qt.AlignRight)

        self.setLayout(controlLayout)
        self.resize(500, 125)

        self.connect(buttonBox, SIGNAL("accepted()"),
                     self, SLOT("accept()"))
        self.connect(buttonBox, SIGNAL("rejected()"),
                     self, SLOT("reject()"))

def main():
    app = QApplication(sys.argv)

    filename = os.path.join(CONFIG_DATABASE_PATH,
                            CONFIG_DATABASE_NAME)
    db = QSqlDatabase.addDatabase("QSQLITE")
    db.setDatabaseName(filename)
    if not db.open():
        QMessageBox.warning(None, "Combo Box Example",
            QString("Database Error: %1").arg(db.lastError().text()))
        sys.exit(1)

    # Would normally be invoked as modal dialog.
    # But for simplicity we use it as the main form here.
    form = EditItemDlg()
    form.show()
    app.exec_()
    del form
    del db


if __name__ == '__main__':
    main()

The main() function is mostly just Qt boilerplate — and code to initialise the database and main dialog window. The EditItemDlg class demonstrates a simple use of QSqlTableModel to populate a combo box.

In the __init__ method of EditItemDlg, we create an QSqlTableModel for the phone_number_type table. We then create QComboBox control, and replace its default model with our QSqlTableModel by calling the setModel() method.

We then call setModelColumn() to tell the combo box which database field it should use as option text to populate the choices in the drop down of the combo box — in this case, the label field of the phone_number_type table.

The result is a combo box widget conveniently populated with choices based on data currently in the phone_number_type table.

PyQt database driver loading issues on Windows after installing PySide

2013-01-24T14:14:00+11:00

Had a Windows machine with a perfectly fine and functioning development setup of ActiveState Python 2.7, QT SDK, and PyQt bindings. Then decided to install PySide. What could possibly go wrong?

There’s always something…

After installing PySide, none of my PyQt SQLite database applications would work anymore (while still using the PyQt bindings). Not even the example programs. I would get a “Driver not loaded” error when trying to initialise the database. However, the PySide database examples ran fine, so looks like installing PySide messed something up in the PyQt installation. After a couple hours of poking around and Googling (there’s hours of my life I’m not getting back!), I worked out what the problem was.

After installing PySide, this is what C:\Python27\qt.conf contained:

[Paths]
Prefix = C:/Python27/Lib/site-packages/PySide
Binaries = C:/Python27/Lib/site-packages/PySide
Plugins = C:/Python27/Lib/site-packages/PySide/plugins
Translations = C:/Python27/Lib/site-packages/PySide/translations

I changed it to:

[Paths]
Prefix = C:/Python27/Lib/site-packages/PyQt4
Binaries = C:/Python27/Lib/site-packages/PyQt4
Plugins = C:/Python27/Lib/site-packages/PyQt4/plugins
Translations = C:/Python27/Lib/site-packages/PyQt4/translations

And once again, I was able to run my existing PyQt database code successfully.

Looks like I’ll have to maintain 2 copies of that qt.conf file, and switch between them depending on whether I’m running PySide or PyQt applications. Not ideal, but at least things are working once again.

In hindsight, I probably should have used the setuptools installation method, possibly inside a virtualenv.

Installing NX Client on Linux

2012-10-17T17:33:00+11:00

So you wanna remotely access a Linux box? From a Windows machine? The Nomachine NX Client looks like a reasonable option.

To install on a 64-bit (Ubuntu-based) Linux, first download the NX Free Edition for Linux. You should end up with something like the following deb packages:

nxclient_3.5.0-7_amd64.deb
nxnode_3.5.0-9_amd64.deb
nxserver_3.5.0-11_amd64.deb

Then:

sudo chmod 755 /usr/lib/cups/backend/ipp
sudo dpkg -i nxclient_3.5.0-7_amd64.deb
sudo dpkg -i nxnode_3.5.0-9_amd64.deb
sudo dpkg -i nxserver_3.5.0-11_amd64.deb

If all goes well, that should conclude the Linux side installation.

All that remains is to download and install the NX Client for Windows on the Windows machine, and a world of stultifyingly laggy screen refreshes awaits.

SVN in a world of Git and Mercurial

2012-08-08T00:00:00+10:00

Oh, so source code for that project I’m working on has been moved from CVS to SVN? Great, thanks for not telling me earlier so I could make sure all my changes were committed first.

Well it is about time we got rid of CVS. But isn’t moving to SVN a bit anachronistic given we have Git and Mercurial (and even Bazaar) nowadays, ready and willing to fulfill all our deepest, darkest version control needs? Isn’t SVN a bit…2002, as opposed to 2012?

No I am not being a smartarse. Just saying, is all. If I was being a smartarse I’d say that SVN is only an improvement to CVS in the same way being poked with a less pointy stick is better than being poked with a really pointy stick, but ideally I’d rather just not be poked by anything.

Oh, so eventually all the projects are going to move to SVN, but we can’t just yet? Because a large legacy project with decades of history was just imported into the one central SVN repository and it blew out the disk usage? So we now have to get a new machine to act as the SVN server?

You realise that with Git or Mercurial, you can pretty much put the repositores just about anywhere, and even move them around pretty much at will? Or even use one of the cloud hosting services?

No I am not being a smartarse. Just saying, is all. If I was being a smartarse I might make some kind of tortured analogy, comparing the switch from CVS to SVN to turning up to a gunfight with a butter knife, sneaking away to a weapons store, walking down the back past racks with all manner of handguns and automated weaponry, selecting a machette, then returning to the gunfight full of misguided confidence. The machettee might be better than a butter knife, but in a gunfight it’s not exactly the best tool for the job at hand. And SVN is to modern version control what a large piece of warm, soggy lettuce is to both grastonomy and flagellation — an entirely unsatisfying experience.

No I do not want to have a meeting to discuss my attitude problem. Merely expressing a subjective opinion.

Oh, and to make matters worse, the build systems for some projects have weird dependencies on the actual SVN revision numbers? Making moving projects or splitting up the central repository tricky at best and insanely complicated at worse? This just keeps getting better and better.

No, I certainly do not want the job of build controller. I’ll shut up now.

With teeth gritted and expectations suitably lowered, I will adjust my workflow to SVN…while silently closing my eyes and thinking of Git…

Some obscure (and not so obscure) SVN tasks

SVN diff without unified diff format:

svn diff --diff-cmd diff -x -b <filename>

Merging cherrypicked changes (in individual files) from trunk to branch:

Check out branch:

svn co svn+ssh://svn/usr/local/svn/project/branches/ver_1_42/ ver_1_42

Find revision number of change on trunk:

svn diff -c 9176 file.c

Change current working directory to branch checkout and merge just the changes in the specific revision from the trunk:

cd ver_1_42/src/sub/dir/
svn merge -c 9176 svn+ssh://svn/usr/local/svn/project/trunk/src/sub/dir/file.c file.c

Rinse and repeat for all files as necessary.

(This worked, but in this case I was only interested in the changeset from a single commit/revision, albeit with many files affected. Might not work as well across multiple revisions?)

Revert changes from one file on new branch (reverse merge from trunk):

cd ~/checkout/project/tags/ver_2_41_3/src/sub/dir
svn merge -r 1449:1448 svn+ssh://svn/usr/local/svn/project/trunk/src/sub/dir/file.c file.c

Canon MP Navigator “Scan Failed” Error

2012-08-07T23:31:00+10:00

It was late. I should have gone to sleep by now. I wanted to go to sleep. I was about to shut down my laptop and get ready to go to sleep. But I remembered I had to scan a document then email it. I should have done this weeks ago. It’s late, but I’ll quickly do it anyway. Shouldn’t take long. The laptop is on, the Canon Multifunction MP800 is right there next to it, thanks to a recent reorganisation of furniture so that all the computer equipment is in one place, and I have the document right here.

How long could it possibly take? (Ha! I had to ask!)

Always far longer than I expect, it seems.

I turn on my temperamental MP800 multifunction printer/scanner and wait for it to complain it’s out of ink.

Yes I know you are out of ink, and I don’t care, because I use the cheap laser printer next to you for my printing needs, to avoid the corporate extortion that is inkjet ink prices, so you have been relegated to just occasional scanning duties. Now hold still while I semi-randomly hold down various combinations of buttons on your control panel until I eventually manage to dismiss your annoyingly persistent errors.

I start up the Canon MP Navigator application on the laptop, which has usually done a serviceable job of scanning documents and saving them as PDF files. I place the document on the scanner. I click the “Scan to PDF” button in MP Navigator. MP Navigator seems to sit and ponder this task for a while, and then a “Scan failed” error message appears. Thank you for that oh so helpful and detailed error message you useless piece of corporate bloatware. What do mean “Scan failed?” The scanner is right there next to you, all plugged in and waiting for your command!

And you were working fine last time I tried this.

At least I think you were.

This actually did get me thinking. I suspect I might have been doing most of my scanning in the past with my old Vista laptop, where everything was working fine. However I was sure that at some point reasonably recently I also did some scanning with my this particular laptop in question — a newer Windows 7 machine. In any case, it certainly had all the right software installed. But I suppose it is possible that while I did install all the required software, I have never actually tried scanning from this particular laptop. I have almost certainly printed from this laptop, but perhaps I have never used it for scanning?

In any case, scanning is now broken.

So much for an early night. Because I Will Not Be Able To Sleep Until I Get This Fucking Thing Working.

I go to the Canon website. I check for newer versions of drivers and MP Navigator. Looks like my system has the latest versions of everything.

I delve into the dark depths of the start menu, looking for any other piece of software that might be able to scan. I do actually find something, and manage to successfully scan an image using a different piece of software already installed on my system. This would indicate that the actual scanner is working, so the problem must be with the MP Navigator software. Unfortunately while I can get a scanned image, I can’t export to PDF with this other software, so I still need to work out what the hell is wrong with MP Navigator.

So I google. Eventually, I manage to find a solution.

I get scanning working by adding the following directory to the user PATH environment variable:

C:\Windows\twain_32\MP800

For those not used to delving into the long forgotten labyrinthine depths of the Windows 7 Control Panel to change an environment variable, follow these steps:

Navigate to Control Panel-->System and Security-->System.
Select the “Advanced system settings” option that’s attempting to hide unobtrusively in the sidebar, which will bring up the “System Properties” dialog.
Go to the “Advanced” tab, and click on the “Environment Variables” button at the bottom.
Pause momentarily to marvel at the simplicity and intuitiveness of Windows.
Select the PATH environment variable from the list, then click the “Edit…” button.
Add the above directory to the list of semi-colon separated directories already there.

Scanning should now work.

After doing the above I was able to scan and save documents as PDF using MP Navigator. Better late than never.

As is often the case with these things, the problem was relatively straightforward to solve once I knew what the problem was. But the weird thing is I don’t remember having this problem, or having to manually configure this environment variable, on my old Windows Vista laptop.

The whole thing reminds me of an old Eddie Izzard routine. “Control-P-print” indeed. Or in this case, “Scan, damn you!”

What license agreement?

2012-06-26T18:56:00+10:00

DEVELOPER:

I’ve create a new MSI installer using WiX, for that legacy front end Windows application. Does the install wizard need to display a license agreement?

MANAGER:

License agreement?

DEVELOPER:

Yeah, the default WiX user interface includes a license agreement page.

MANAGER:

What’s on the license agreement page at the moment? Boiler plate text?

DEVELOPER:

No idea. Never read it. It’s a license agreement page. Just ignored it and clicked next…

Scripts for batch renaming of files to add a date prefix

2012-03-29T00:00:00+11:00

A financial institution of which I am a customer (which shall remain nameless) frequently (like almost every weekday!) emails me PDF statements. I rarely need to refer to said statements, so I could just configure my email client to automatically filter them away into some folder and not give them the slightest consideration. But occasionally I might need to check something in one of these statements, usually around Tax Return time, and usually something from many months/years past. Sifting through PDFs stored as email attachments is not ideal. Therefore, I download the statements, and store them in a Dropbox folder for safekeeping, in case The Tax Man ever enquires about their numerical specifics. This makes things easier to find if I need to look something up. Or it would if the financial institution in question had shown more foresight in the naming convention of said statements. They have filenames like:

29 March 2012 Statement.pdf

I probably shouldn’t complain about this. At least there is an unambiguous and — more importantly — consistent date component as part of the filename. Including the year. That’s certainly better than having each and every statement file called “Statement.pdf”.

Still, ideally I’d want the date to be of the format YYY-mm-dd, so that a statement for a particular date is easier to find, given a directory full of these files.

But I’m certainly not going to bother renaming each file manually. Especially since I’m a coder. I’ll write a script to do this instead. But if I’m going to write a script, might as well also automate the step of copying these PDF files from the “Downloads” folder, where the attachments initially end up, to their ultimate destination in my Dropbox.

So here’s an implementation of this as a Bash script:

#!/bin/bash

FROM_DIR=/home/rob/Downloads
TO_DIR=/home/rob/Dropbox/Docs/statements

shopt -s nullglob
for f in ${FROM_DIR}/*Statement.pdf
do
    date_part=`basename "${f}" | sed -e 's/ *[S|s]tatement.*pdf//;'`
    new_date=`date --date="$date_part" "+%Y-%m-%d"`
    suffix=`basename "${f}" | sed -e 's/ //g;'`
    new_filename="${new_date}_$suffix"
    if [ -e "${TO_DIR}/${new_filename}" ]; then
        echo "`basename "${f}"` not copied. File already exists: ${new_filename}"
    else
        echo "Moving `basename "${f}"` to ${new_filename}"
        mv -i "$f" "${TO_DIR}/${new_filename}"
    fi
done

And here’s the same thing as a PowerShell script, so that I can download these statements from whatever machine I’m currently working on, be it Linux or Windows:

$from_dir="C:\Users\Rob\Downloads"
$to_dir="C:\Users\Rob\Documents\Dropbox\Docs\statements"

$files = get-childitem $from_dir\* -include *Statement.pdf
foreach ($file in $files) {
    if ($file.name -match "^\d\d [A-Z]{3} \d{4} Statement.pdf$") {
        $date_part = [regex]::Replace($file.name,(" *Statement.*\.pdf"),"")
        $new_date = get-date $date_part -format yyyy-MM-dd
        $suffix = $file.name -replace " ", ""
        if ($?) {
            $new_file = $new_date.toString() + "_" + $suffix
            "Moving " + $file.name + " to " + $new_file
            $new_path = $to_dir + "\" + $new_file
            if (Test-Path  ($new_path) -PathType leaf) {
                "File already exists in destination!!!"
            }
            else {
                move-item $file $new_path
            }
        }
        else {
            "Error processing " + $file.name
        }
    }
    else {
        "Skipping " + $file.name
    }
}

Programmatically playing with AmiBroker watchlists

2012-03-03T15:00:00+11:00

If you’re an AmiBroker user, you may have spent much time painstakingly creating and editing watchlists of stocks, with all the repetitive pointing and clicking that entails, to get them just right. With that done, and your watchlists for each and every database now organised to perfection, you might now be wondering: “How can I fuck it all up completely in the minimum amount of time?” Got Perl installed? Well then have I got the perfect Perl script for you!

(Because nothing has the power to mess up perfectly good data like a poorly written Perl script! Well, OK, that’s not really true. Regardless of programming language, any determinedly incompetent programmer could no doubt write a script to neatly screw up data and/or sanity with a simple command line execution. Sadly, sometimes this even happens to the best of us. Which is a roundabout way of saying having a backup of any data before running experimental scripts to mess with that data might be a good idea. I should also point out what follows is intended mostly as an attempt to get my own thoughts in order, rather than being any practical use to anyone else, as I realise the potential audience of Perl-programming AmiBroker users is small to non-existent. This is merely an idea I’ve been messing with.)

My motivation for this script (and this post) is a frustration with the unnecessarily repetitive pointing-and-clicking that many GUI applications inflict on their users (namely, me). Charting application are especially prone to this, and given their visual nature this might be somewhat unavoidable. But it seems to me even well designed charting applications — and I consider AmiBroker to be an excellent example of the genre — have their user interface annoyances. Or maybe I’ve just been spending too much time in the command-line world of Linux lately. Or maybe I expect things from charting applications that no one else does. Or maybe I do things with watchlists that no one else wants to do. But in any case, I have yet to find a charting application that provides some way to graphically maintain “watchlists” of stocks without making me pine for a command line alternative.

The annoyance in this case some might consider a minor one, but it annoyed me enough to make me turn to two things that I often find even more annoying — Excel and Perl.

Here’s the thing. Not counting databases for indices, I have 4 AmiBroker databases set up — one ASX database for Australian stocks, and three US databases, one for each of the major exchanges (NYSE, NASDAQ and AMEX, though can’t remember the last time I looked at an AMEX stock chart). Since each database has its own separate sets of watchlists, that means 4 sets of watchlists to maintain. (Note I’m talking about the names and order of the watchlists for each AmiBroker database, not the individual stocks, which obviously will be different for each exchange).

In an effort to keep on top of the more mundane record keeping tasks of trading, while spending as little time as possible doing so, I have been becoming more militant with maintaining watchlists. In some cases, I might want the same semantic watchlist across all 4 database — for example, a watchlist for currently open trades, so I can quickly scroll through charts of all my current trades with minimal keystrokes/mouseclicks. In other cases, however, I might have a watchlist that is only specific to ASX, and not to the US exchanges — for example, a watchlist for Australian resource companies. So in general, I want the AmiBroker databases for the 3 US exchanges to have identical watchlists, but the ASX one to be a bit different. Of course, I can do this manually. By pointing and clicking. But I am a coder. That’s not how I roll.

So I wrote a script.

Fortunately, AmiBroker uses a simple text-based watchlist format that lends itself to programmatic hacking. Within each AmiBroker database directory, there will a subdirectory for “watchlists”. Within this directory, there will be a file called index.txt. This is a simple text file, where each line is the name of a watchlist, specific to the database in question. There can apparently be any number of watchlists. The actual stocks that comprise each watchlist are stored in separate data files — for each entry in index.txt, there should be a corresponding text file with a .tls extension, which is simply a list of stock codes comprising that watchlist. But these tls files don’t concern us for now.

Of course, all this is current at the time of writing, and subject to change. But with keen awareness of the hackiness of it all, I created a spreadsheet with the names of watchlists, arranged how I wanted them, then wrote a script to export that spreadsheet into AmiBroker watchlist index files. This feels hacky, and will obviously break if future versions of AmiBroker handle watchlists differently, but for now it achieves its intended purpose of saving me some tedious mouse clicking. It means when I want to add, delete, or rearrange ordering of watchlists, I can far more quickly edit the spreadsheet and run the script. (Before backing up the AmiBroker databases first, of course! And making sure AmiBroker isn’t currently running at the time, or Bad Things Could Happen.)

For anyone who’s interested, here is the script:

#!/usr/bin/perl
use strict;
use warnings;
use Spreadsheet::ParseExcel;

#
# Check and extract command line arguments -- or setup OS-specific defaults.
#
my $g_infile = "AmibrokerWatchlists.xls";
my $g_dir = "watchlistdb";

if($#ARGV >= 1) {
    $g_infile = $ARGV[0];
    $g_dir = $ARGV[1];
}
else {
    die "Usage: $0 <infile.xls> <outdir>\n";
}

#
# Column 0 is index number of watchlist.
# Column 2 is ASX
#        3 is NYSE
#        4 is NASDAQ
#        5 is AMEX
#

#
# Indexes of columns in spreadsheet, and corresponding exchange names.
#
my $CI_INDEX    = 0;
my $CI_ASX      = 2;
my $CI_NYSE     = 3;
my $CI_NASDAQ   = 4;
my $CI_AMEX     = 5;

my %exchange;
$exchange{"ASX"}        = $CI_ASX;
$exchange{"NYSE"}       = $CI_NYSE;
$exchange{"NASDAQ"}     = $CI_NASDAQ;
$exchange{"AMEX"}       = $CI_AMEX;

my %watchlist;

unless (-d $g_dir) {
    mkdir($g_dir) or  die "Can't create $g_dir directory: $!";
}

my $parser   = Spreadsheet::ParseExcel->new();
my $workbook = $parser->parse($g_infile);
if (!defined $workbook) {
    die $parser->error(), ".\n";
}

#
# Extract data from spreadsheet.
#
my $sheet = $workbook->worksheet(0);
my ($cell, $wname, $windex);

my ($row_min, $row_max) = $sheet->row_range();
my ($col_min, $col_max) = $sheet->col_range();

if ($row_min + 5 >= $row_max) {
    die "Invalid input file: not enough rows!";
}

foreach my $row ($row_min+5 .. $row_max) {
    #
    # Get value of first column.
    #
    $cell = $sheet->get_cell($row, $CI_INDEX);
    next if (!defined($cell)); # Skip row if the first column is blank
    $windex = $cell->value();

    print "$windex:\t";

    while (my ($e, $i) = each(%exchange)) {
        $cell = $sheet->get_cell($row, $i);
        if (!defined($cell)) {
            $wname = "List$windex";
        }
        else {
            $wname = $cell->value();
            $wname =~ s/^\s+//;
            $wname =~ s/\s+$//;
        }
        push(@{$watchlist{$e}}, $wname);
        print "$wname ($e)\t";
    }
    print "\n";
}

print "\n=====\n\n";

#
# Generate output files.
#
foreach my $k (keys %exchange) {
    if (-d "$g_dir/$k") {
        # Delete any existing TLS files in the directory.
        my @existing = <$g_dir/$k/*.tls>;
        for my $tls (@existing) {
            print "Deleting: $tls\n";
            unlink($tls);
        }
    }
    else {
        mkdir("$g_dir/$k") or  die "Can't create $g_dir directory: $!";
    }

    open(OUTPUT, ">", "$g_dir/$k/index.txt") || die "Can't open output file: $!";

    foreach my $l (@{$watchlist{$k}}) {
        print OUTPUT $l, "\n";
    }

    close(OUTPUT);
}

(View as a gist here)

The idea here is fairly simple. The script generates index files for each database based on the first worksheet in the spreadsheet file. It outputs one index file for each database (i.e. stock exchange), placing each one in a subdirectory with the database name. It doesn’t directly modify or overwrite any files in an AmiBroker database (I’ve got another script for that!), nor does it attempt to even mimic the proper directory structure. All it does is output index.txt files which can then be (manually, and carefully!) copied to the appropriate subdirectories within the respective AmiBroker databases.

The first 5 rows of the spreadsheet are ignored, so they can contain any arbitrary header information for human consumption.

The first column (from row 6 onwards) must contain either a number, or be blank. If it’s blank, that entire row is ignored. If it has a number, then this number represents a watchlist index number. Column B is ignroed. Columns C to F contain the names of watchlists for databases ASX, NYSE, NASDAQ, AMEX (in that order). So if column A has an index number, then columns C to F in that row represent the watchlist names with that index number, for all the respective databases.

Since I like to have all the US watchlists identical, I could have made the script more specific and just had 2 columns — one for ASX, and one for US databases, but I decided to implement the more general case, even if it means having to do a fill in Excel after editing US watchlists.

Goodbye crappy overpriced overhyped charting application, hello AmiBroker

2012-01-28T16:15:00+11:00

So I’ve had this idea for a post/rant knocking around in my head for ages, but been unsure whether I should actually write it. For one thing, it would essentially amount to a review of software I no longer use, haven’t used in a long time, and have no intention of ever using again. However, I do use one of it’s competitors, which I consider a much superior piece of software, and which I will happily continue to use. Like all software, however, it is not perfect, and there are some things about the user interface I find a bit frustrating, though no where near as much as the former. So just in case at some point I happen to write something that might be remotely critical of the latter, I should at least bag the shit out of the former first, especially since it is what made me appreciate the latter so much — and as such provide some context for any future ramblings about trading software. Not to mention it might be good to vent. So here goes.

Once upon a time, years ago, I decided to dedicate an increasingly large chunk of my increasingly smaller free time to trading the stock market. That was probably my first mistake. So I went searching for software tools to assist me in this misguided quest. Specifically, I wanted a charting program. Being a beginner trader, my needs were fairly modest, but I thought I had a good idea of what I wanted — just the common technical analysis indicators, and ability to search a stock price database based on common technical criteria. My research led me to one piece of software which seemed popular and well regarded — something which now, having used the software in question, I find completely mystifying. I won’t mention the name of this charting application, because it is an abomination against all that is good and holy in user interface design and software quality. Let’s call it the “Charting Application That For Usability Crappiness The Name Of Which Is Not To Henceforth Ever Be Uttered or Mentioned”, or CATFUCTNOWINTHEBUM for short. OK, that’s not very short. It’s actually twice as many letters as the real name of the software. So how about we just call it the Charting Application That Possesses Pathetically Poor Usability (CATPPPU), and just agree never to mention its name?

Well, OK, to be fair (kind of), if all you want to do is look at the occasional chart, CATPPPU does the job. (Though with the current state of computer hardware/software, drawing a 2 dimensional chart of stock prices is hardly a taxing task.) Where the software really fell down is with the more advanced functionality around its charting core — like searching through stock price data, and manipulating and maintaining lists of stocks.

The user interface was like something out of a bad 90’s shareware application. It was full of small, cramped, non-resizable dialog boxes — one of my personal pet-peeves of user-interface design. What’s worse, these undersized dialogs were either crammed with controls, or were filled with one or more table widgets full of data, which, once populated, necessitated constant scrolling of sliders to see the data. I felt this was just inexcusable for a mature software application like this, since one of its main features — and reason for existing — is allowing users to generate and work with that data easily! And the ultimate frustration of not having resizable dialog boxes in an interface of a charting application for traders, is that if any class of computer user (apart from developers) are likely to have large screens and even multiple monitors, it’s traders! Nowadays large LCD screens and multiple monitor setups are relatively common for desktops machines, but hardcore traders have been using multiple flat screens way back when most people were still stuck with a single bulky CRT. So I would regard it as quite a serious deficiency in a piece of software supposedly aimed at serious traders, to cram all its advanced functionality into small, cramped dialogs that can’t be resized to take advantage of the larger screen(s) the target end user is likely to have!

Then there’s the functionality for creating and managing “watchlists”, such as it was. It seemed this part of the user interface was modelled loosely on early versions of Internet Explorer’s “Favourites” feature. Modelling anything on Internet Explorer features is rarely a good idea.

The whole “watchlist” thing was actually my main reason for buying an overpriced charting application in the first place. The workflow I wanted to achieve is to to create a list of stocks, usually by a search based on technical analysis criteria (e.g. moving averages, etc), then quickly scroll through the resulting list, having a quick look at each one, and discarding those I didn’t like the look of. The remaining stocks on that list would then be worthy of deeper analysis. I didn’t think that was too much to ask of a modern charting application. But if there was a way to easily achieve this workflow, I couldn’t work it out, despite going through the manual and googling. I had to resort to copying results of searches into Excel, then flicking back and forth between the two applications to get anything even close to the workflow I wanted.

Even worse, the minimal watchlist functionality this software did have was poorly implemented, buggy and slow as all fuck. The program would frequently crash while I was trying to manipulate watchlists. And even when it didn’t, deleting one or more stocks from a list would make it freeze up and furiously thrash the hard drive for minutes at a time. To delete one item from a list! On a modern PC! Seriously! We’re not talking about huge data structures here — just lists of stock codes, or some kind of references to data files on a local hard drive. What kind of convoluted internal data structure were these watchlists using that it took up to 10 minutes of thrashing the hard drive to delete one tiny item of data from a list? What could it possibly be doing for so long? As a developer, that indicates the watchlist functionality was phenomenally poorly implemented. What happened? Did they get the work experience kid to work on that part of the program? Without supervision? This kind of thing is seriously amateur hour — and I paid a thousand bucks for this shit!

I persevered for a while, but then it came time to get a new PC, with a newer version of Windows — Vista. Not surprisingly, the version of CATPPPU I was using was not compatible with Vista, and there was no fucking way I was going to spend more money to upgrade this shitty software. So I went looking for alternatives. I ended up switching to AmiBroker, which turned out to be better in just about every way. Not only that, it was cheaper, and worked across any version of Windows I cared about.

Now, I don’t really like getting all philosophical about software development. The people who do wax lyrical about software/websites/apps as “art” are usually either trying to bullshit their way into getting venture capital funding, or a venture capital douchebag trying to justify why he just invested millions of dollars in a company that isn’t making any money and will likely never ever make any money. Or just Apple fanboys.

But there is undeniably an aspect of art involved, both in the process of coding, but also in the more visible area of user-interface design. When watching a good movie or experiencing a great band perform, it is often obvious from the start that the people involved really know what they’re doing, they’re on top of their game, that they have crafted their art with skill and attention to detail. In the same way, it can be obvious when a piece of software has been well crafted by people who put some thought and attention into how it should work, as opposed to just a bunch of careless cubicle monkeys phoning it in. That is the difference between AmiBroker and CATPPPU. AmiBroker and CATPPPU are both applications that set out to fill a particular need and provide a specific set of functionality. Yet, while CATPPPU does kind of get the job done, it does it clumsily — it’s basically a great example of lazy software, as the user inferface it provides is about as basic and minimal as possible, and certainly without style. They obviously threw a bunch of features in there, while doing as little work possible to expose those features to the end user. They certainly didn’t think about the user experience in using those features. AmiBroker, by contrast, is an example of the same basic idea done right. It’s not perfect, but it gets the job done with a clean, modern user interface.

I actually found myself at an “educational” trading seminar last year — where “educational” in this context means “shameless promotion of trading products and services but with a few authors of second rate trading books throwing up some charts every now and then to provide a thin veneer of credibility, before shamelessly spruiking their own second rate books”. At one point some guy from the company currently distributing CATPPPU in Australia got up and bullshitted about what a great product it is, while I sat down the back and tried to refrain from laughing hysterically. He hooked his laptop to the projector to provide a demonstration. I’d completely given up on CATPPPU by this stage, and had long since switched to AmiBroker. Watching him work CATPPPU on the projector I felt even more justified in my decision, as it reminded me of some other user interface failings of the software that by then I had forgotten.

He demonstrated how to run scans on the database, but to do so involves selecting which stocks (or subset of stocks) to search through. This is done though fairly standard windows open/browse dialog boxes (non-resizable, if I remember right?). But I don’t think there’s any way to select a directory (and it’s entire contents). You have to go into every directory, press CTRL-A to select every data file in it, wait while the software freezes up for a bit if there are lots of files in the directory (which there usually will be), click open, then repeat the process for each directory (of which there will be around 26 if you happen to have the data files sorted in A-Z directories, a default of many data suppliers because the proprietry file format it uses has an arbitrary limit of 2000 data files per directory!). And you have to do this every fucking time you want to run a scan — unless you manage to painstakingly, and with much hard drive thrashing, create a watchlist without the software crashing on you. Not surprisingly he glossed over this point during the demonstration.

The real kicker for me, however, was when, in an attempt a levity, he pointed out the most common support call they get from customers is right after they install and run the software for the first time. And it’s always the same question:

“What do I do now?”

Because the software starts up with essentially a blank window with only a menu bar at the top. Nothing to indicate how to open a chart, or where to load data from. He then went to great lengths to reassure everyone that their support department is ready and waiting for their calls, and will be all to happy to help out if anyone encounters problem during or after installation. This for me was more proof of how hopelessly broken this software was from an user experience point of view. He shouldn’t have had to make that assurance, and ideally the software should do more to guide users what to do when first run.

Another telling example of the difference between the art of making quality software with care for the end user experience and attention to detail, and software just hacked together for the money. Don’t get me wrong, CATPPPU does what it claims to do, but in my subjective opinion, it doesn’t do it at all well. I think just about everything in its user interface could have been implemented better. AmiBroker is proof of that.

Goodbye Unity, hello once again KDE

2011-12-06T19:23:00+11:00

Goodbye Unity. I gave you a go, I really did. You are a bold move in the world of Linux, which is good to see. But you’re just not working for me. You’re OK, and no where near as bad as some have made you out to be, but as a development machine desktop, I find you’re less than ideal. So I’m going back to KDE. Which is not to say that KDE is perfect, just the lesser of desktop environment evils. And I do appreciate how integrated and polished you are in a lot of ways Unity, especially visually. This is in contrast to the GTK based applications like Firefox and Thunderbird on KDE, which end up looking annoyingly out of place and ugly. But unless I find workarounds, I will put up with the ugliness, because since I spend most of my time doing fairly low level back end coding, I care little for fancy GUIs and desktop environments, and would just prefer they stay out of my way as much as possible. But goddamit Firefox you are looking hideous on KDE compared to Unity…

UPDATE: Installing KDE on top of Ubuntu (though the package manager), as opposed to Kubuntu, might have contributed to the above ugliness. I’m not sure if recent Kubuntu distributions have the same quirk, but I notice Firefox looks fine on a home PC currently running Mint 13 KDE.

Enable Apache userdir

2011-11-07T20:37:00+11:00

If I’m using a Linux box as a development machine, I usually like to have Apache installed with userdir enabled, even if I’m not necessarily doing web-development, just in case I need to play with something webby/internety.

Enabling the userdir module on Unbuntu and its derivatives is usually something like:

sudo a2enmod userdir

If required, further tweaking of userdir configuration can be done by editing:

/etc/apache2/mods-enabled/userdir.conf

Then restart apache:

sudo /etc/init.d/apache2 restart

After this, all manner of HTML-based experiments can be installed or inclubated in the relative safety and convenience of a user’s home directory at ~/public_html/, which should be accessible on http://localhost/~username/.

Just make sure all subdirectories in path to public_html have read and execute permissions.

Make Firefox open new tabs at far right

2011-11-03T19:43:00+11:00

Looks like I’m gonna have to do this every time I do a fresh Linux install — configure Firefox to open links in new tabs with the new tab positioned at the far right. The way I like it.

Type the following into the address bar:

about:config

If a warning shows up, promise to be careful.

The option that needs to be configued is:

browser.tabs.insertRelatedAfterCurrent

Easiest way to find it is enter it into the search box. Set it to “False”.

Now tabs will open where I expect them to open.

Batch rename of filenames to lowercase

2011-10-10T18:59:00+11:00

I must have done something terrible in a past life, for as part of my day job I have to maintain software written in Visual Basic. Version 6. Yes, version 6. And it’s 2011. I’ve checked my watch. Twice.

The Visual Basic abomonation is only a small part of a much larger Unix-based client-server system — the VB applications are the GUI front end. It’s interesting that even when doing Windows development, remarkably often I find the need to do something that would be easily accomplished on a Unix box, but is innordinately difficult to do on a Windows machine. Even as a (power) user, I’m increasingly finding I’m more productive, not to mention less annoyed, with the user experience on my Linux box at work, compared to my Windows 7 machine at home. (Although both Linux and Windows GUI environments have things that annoy me.)

In fact, pretty much the only compelling argument I’ve heard to justify the relative expense of purchasing a Mac is that underneath the eye-candy and consumer friendly front-end of OS X, there lies the power of Unix, for those that might want or need it. Although to be fair, PowerShell does go some way towards solving simple cases of where one might need some command-line goodness. But that would mean having to learn PowerShell.

And so once again I find myself in a situation where Windows unnecessarily gets in my way, but thankfully I have access to a Unix system, which allows me to easily fix the problem at hand relatively easy. In this case, the problem is that one of the many, many, many annoyances I am faced with is Visual Basic’s tendency to change the case of filenames that are part of the project. This is not normally an issue in the case-insensitive world of Windows. But these pieces of Visual Basic detritus are parts of a mainly Unix system, so therefore it is on the Unix side of things where the source code version control is handled, and where what should be a simple act of committing code changes inevitably become a 3 act tragedy, often involving much time working with Emacs ediff mode.

Fortunately, doing batch renames on Unix/Linux is fairly straightforward thanks to the rename command. So to ensure all files in the current directory with a “.frm” extension have all lowercase filenames:

      rename 'y/A-Z/a-z/' *.frm

Easily fixed on just about any Unix/Linux system, thank you Larry Wall. Try doing that on any out-of-the-box Windows installation — without the tedium of endless pointing and clicking. Or learning PowerShell.

Cold And Grumpy: Why hast thou failed me my modern mod cons?

2011-09-04T16:43:00+10:00

It seems to happen in threes. All of a sudden my digital TV reception has gone to hell. Being without TV I might consider a minor annoyance, in a world of online streaming of TV shows (and *cough* BitTorrent *cough*). But my ADSL is constantly dropping out too. And my central heating just broke down. So I am without both TV and internet. And I’m cold. Needless to say I’m fucking grumpy.

The ADSL I’ve been having problems with on and off for a very long time. It’s no doubt a result of Telstra’s crappy copper infrastructure. The central heating unit is old, and has been breaking down with remarkable regularity every year. The TV reception problem, however, is new. I have not had a problem with this before, but all of a sudden it is so bad it’s pretty much unwatchable, all channels are totally garbled. Switching to analogue, the picture is watchable, although with interference lines moving up and down the screen and regular audio glitches. What the fuck? Nothing has changed in my TV setup lately.

When I switched to digital, I replaced all the antenna cables. It seems the old ones were made from a material completely incapable of transmitting digital data reliably. Much like Telstra’s copper network.

So I have new, properly shielded cables, along with a powered splitter. It’s all been working fine for months. Nothing has changed. What is going on?

I checked and double checked all the cables and connections. I moved things around. I unplugged everything except the TV. I disconnected the splitter and ran the TV antenna cable straight into the socket. Nothing made a difference. The TV was reception still stuffed. I looked at the TV antenna outside. As far as I could tell, everything was still connected, cables and connectors intact, nothing looked obviously wrong.

What could I do? Call someone? Who do I even call for TV reception problems like this?

I thought I’d wait a few days and see if the problem would go away on its own. (An eternal optimist, aren’t I?) It didn’t. The TV reception problems continued for days. Cold days. With no heating. And only intermittent internet.

I was also wondering what to do about the heating. Looks like it’s time to replace the heating unit, but with the colder months behind us for the year, might wait till next year before deciding whether to replace or repair the damn thing. But in thinking about this, it occurred to me that the TV reception went to hell on the same day the central heating stopped working. Coincidence? Is there such a thing as coincidence when it comes to technology?

I went outside, around the side of the house, to the central heating unit. I opened the front panel and had a look inside the . The unit was so old it still has a pilot light. Apparently modern units don’t have them. Not surprisingly, the pilot light was out. As a result, the “electronic pilot relighter” was sparking at regular intervals, trying to relight it, but in vain, as most likely something somewhere else had broken down, preventing the pilot light from actually lighting.

I flicked the switch to turn off the automatic pilot relighter, then went back inside and had a look at the TV. It was back! The pristine TV reception is back, in all its crisp digital glory. So that is what had happened. The central heating had broken down, causing the pilot light to go out, unable to be relighted. But that didn’t stop the automatic pilot lighter constantly generating a spark to try and relight the pilot light, and this spark must have been causing massive interference with my digital TV reception. Turning off the automatic pilot lighter fixed the problem.

So that’s one problem resolved.

Now if only I could fix my ADSL problems by simply flicking a switch.

JasperReports Font Foibles

2011-08-30T22:51:00+10:00

One moment I’m thinking JasperReports is great, it’s exactly what I need, it’s making things so easy and my job/life/world is all the better for it. The next moment I find I’ve overlooked/forgotten/misunderstood some arcane aspect of it and everything degenerates into a horrible, broken, useless mess, and I’m thinking why am I bothering with all this when it just seems to be an exercise in endless frustration and misery? Kind of like some past relationships. It’s as if we’re dating.

The latest frustration is fonts. After much time spent creating a particularly “dense” report, with lots of fields, lots of data, and much time spent meticulously tweaking so that everything lines up and is spaced out appropriately while still fitting in all the data and displaying properly, I upload the final project to the production server, only to find this particular report no longer displays properly — some of the fields are truncated. WTF? I checked this. Many times. It was all working properly on my machine. What’s different between my machine and the production machine? Java and JasperReports versions are the same. Turns out the production machine is running RedHat Linux, while I am developing on Ubuntu. Looks like it’s a fonts thing. I was using Java logical fonts for the reports. Seems there is no guarantee even between different Linux distributions that these fonts will map to the same (size) physical font. And seems there are more hoops to jump through to be absolutely sure of getting the right font in a PDF report.

Generating ssh keys

2011-08-17T18:57:00+10:00

To set up ssh to allow logins without having to type a password, first generate keys on the local machine:

ssh-keygen -t rsa

Copy (securely) the freshly generated public key (id_rsa.pub) to the remote machine.

Append the contents of id_rsa.pub to the ~/.ssh/authorized_keys file in the home directory on the remote machine.

Finally, if necessary:

chmod 600 ~/.ssh/authorized_keys

The above procedure will also stop SVN from constantly pestering for a password when checking out a repository. (For those of us unfortunate enough to still be using SVN.)

Generating gSOAP stubs in C

2011-07-20T23:09:00+10:00

I think I might have enjoying my recent foray into the weird world of web services and SOAP had it not felt so much like trying to fit a square peg in a round hole. Reverse engineering a communication protocol that has its roots in RS-232 serial coms was bad enough. But then to work out how to send that data over a web service interface defined by a third part…while maintaining backwards compatibility…in C…let’s just say it was even less fun that it sounds.

gSOAP was used to handle the SOAP stuff, and turned out to be easiest and most reliable part of the whole project…relatively speaking. Because this isn’t a scenario the gSOAP documentation emphasizes (although the info is in there, sortof), here are some crib notes.

So you have some xsd and wsdl files from a third party that specify a web service interface. And for better or worse, you want/need to work with straight C code, as opposed to C++.

Assuming wsdl and xsd files have been downloaded to a local directory, create a “typemap.dat” file in this directory and add a namespace prefix. For example, add a line similar to:

thirdparty = "http://xml.thirdparty.com/WS"

Then use wsdl2h to create the header file:

wsdl2h -c WebService.wsdl

This creates WebService.h.

Then generate the code stubs using soapcpp2:

soapcpp2 -c -C -x -pfilenameprefix WebService.h

The -c options means generate C code, while using the -C option generates only client side code.

Move Gnome 2 window controls to right

2011-07-14T22:18:00+10:00

I’ll give Unity a go one day, but for now I’m persisting with Gnome 2 on my work PC, because I have it set up with the GUI staying out of my way as much as possible, so for now it is the path of least annoyance.

At least now that I have worked out how to fix one of Ubuntu’s annoying allusions to OS X of having Gnome position the minimize, maximize and close buttons on the left corner of window borders. Like most people, I still have to use Windows, and I see no need to have to mentally acclimatize to this arbitrary difference between the 2 operating systems I regularly use.

To change the position of the window controls to the right, run the Configuration Editor by pressing Alt-F2 then typing gconf-editor.

Navigate to:

apps/metacity/general

Find the button_layout setting.

Change the value to:

menu:minimize,maximize,close

Emacs search and replace newline shenanigans

2011-07-05T20:32:00+10:00

I can’t believe I’ve never had to do this before: replace a string of one or more characters in a buffer with a single newline. But how to tell Emacs the replace text is a literal newline? The obvious escape characters like \n don’t work. After googling, turns out the answer is deceptively simple:

      C-q C-j

Not exactly intuitive, but kinda sorta makes sesnse in an Emacsian way. Weird how I’ve never had to do this before, but today I needed to do this twice!

First Post

2011-07-02T16:23:00+10:00

So you know how sometimes when you’re coding or doing some sysadmin kinda thing to try and get the technology to do something you want it to, preferably in the way you want it to do it, and you figure it’s probably something someone else has tried to do in the past and didn’t know how so the question has already been asked on a forum or possibly preemptively answered in a blog post, so the answer could be out there somewhere on the wide web world, so you google and sure enough you find a web page or blog post or Stack Overflow question that provide exactly the information you’re after. Joy and happiness! So you manage to get the technology to do what you want it to do in a manner more or less approximating the way you want it done. Then 6 months later you have to do the same thing again but you have forgotten how to because it’s not something you need to do all that often, so once it was done and working well enough it was out of sight and out of mind and you didn’t make any notes on how you managed to get it working because you didn’t think you’d ever have to do anything like that ever again but that’s exactly when fate and the universe at large and the Gods of Irony conspire to work their perverse mischief and lo and behold here you are again vainly trying to remember how the hell you did that thing to make it work last time, but to no avail, so you end up having to google for it all over again….then 6 months after that you…

No? That’s never happened to you?

Well then, you have totally stumbled across the wrong blog. Move along now. Nothing for you to see here.

I figure I should write some stuff down, if for no other reason that I might want to refer it later on at some point, and do it in blog form on the off chance there might be someone out there googling hungrily for the same morsel of information.

Hence, my geek blog. A place for my brain dumps, and random rambling, and hopefully something of some interest to someone out there on the interweb.

Hopefully with shorter sentences than this first post.

voidynullness

The Terrible Turnbull and LNP NBN legacy

Wayward Windows

Related Posts:

Nexus redux

Related Posts:

LG refuses to honour warranty after my Nexus 5X develops bootloop issue

Related Posts:

Death by bootloop of my Nexus 5X

Related Posts:

VoidyBootstrap version 2.1 released

Related Posts:

Upgrading to Pelican 3.7

Pelican changes

Pelican site configuration

Related Posts:

Android Tabs: How to use TabLayout

Related Posts:

Android Tabs: Using PagerTabStrip with ViewPager

Related Posts:

Big Government Puppies

Related Posts:

Mandatory protocol analysis to the node

Related Posts:

How to add a list of recent posts to a Pelican blog

List of Recent Posts

Now With Added JSON

Fail Faster

What’s New In VoidyBootstrap Version 2

Overview

Sidebar Changes

Custom CSS and Javascript

Optional Templates removed

New Custom Includes

Conclusion

See Also:

Related Posts:

Smooth scrolling using jQuery

Floating scroll to top button

Laggy Lollipop: How to clear the cache partition on an Android Nexus 7

How to wipe the cache partition

The Definition of Metadata

Related Posts:

Mandatory data retention: futile and dangerous

Related Posts:

Who cares about data retention anyway?

Related Posts:

Electronic Graffiti

Related Posts:

The Who Hacked Sony Story

Hello 2015, and goodbye online freedom and privacy

Related Posts:

Upgrading to pelican 3.5

Related Posts:

Metadata karma

Related Posts:

We all have something to hide

Related Posts:

They’re coming for us and our metadata

Related Posts:

To An Australia That Cares

Related Posts:

Piracy Propaganda

How to install a Python development environment on Windows

Python

Setuptools

pip

virtualenv and virtualenvwrapper

Qt and PySide

How to turn a Windows box into a development machine

Browsers

Document Viewers

Dropbox

Backup

Emacs

Notepad++

Office

Python

Java

Development Tools

`registration_form.html`

`registration_complete.html`

`activate.html`

`activation_complete.html`

`activation_email_subject.txt`

`activation_email.txt`