Execution Unit

How I setup Subversion to work with Unreal.

2024-08-21T00:00:00+01:00

Overview

I have used Perforce, Subversion (SVN) and Git with Unreal on AA Game and Indie Games. I feel that Subversion is often overlooked as a source control solution for small to medium teams. Subversion has fallen out of favor and is no longer cool. It definitely earned a reputation for “screwing up branching” but this isn’t an issue for Unreal projects. Subversion is fast, stable and reliable.

This is a blog to help you IF you are interested in an alternative to Git+LFS and Perforce+$$$. If you’re not interested that is fine.

It’s not just me who thinks Git+LFS is bad. Ari Arnbjörnsson also thinks it kinda sucks for Unreal development.

Summary

For small hobby projects Git/GitHub are fine. Ignore this, carry on using Git.

For AAA just use Perforce, you’ve got the money.

For film or architecture/visualization, just use perforce as your files are likely way to big and your staff less open to change.

For medium to AA games I’d argue Subversion is fine. Not perfect, but fine. For teams between 5 and 50 members it fine. For 100+ you’re definitely in the Perforce territory and outside my experience. You might be fine. I’d hazard a guess that you would be fine but this is outside of my experience.

Why

Opinions, you don’t have to agree with me.

Git+LFS is slow

It’s not terrible but it’s slower than Perforce and Subversion. Locking is also not perfect and I found it failed frequently. I do not consider Git+LFS to be mature for the thousands of opaque binary files that Unreal uses. I do not consider Git+LFS locking to be mature for medium sized teams.

Git+LFS doesn’t scale as well as Perforce and Subversion.

Git stores the whole repository over time locally. This is great because you can switch branches trivially, experiment more easily and develop powerful workflows. Alas it means every binary file exists in all it’s forms over the life of the project locally. Even a medium sized Unreal game will be 100GB of live assets and nanite is making this look small. After a few years for a small team of 10 this can result in a dataset nearing 1TB of data. Disk space is cheap but it’s all just noise and slows Windows and git down.

Git’s primary innovation of it’s branching strategy and branching is not used by most Unreal devs because locking files is integral to the workflow of Unreal Editor.

A AAA game will easily have 1TB of live assets and a total dataset can easily be 50TB in size. For this scale Perforce really is the only product in town… but most people will not be working on a game that large.

Perforce

This is industry standard. This is a pointless phrase and tells you nothing of whether it meets your needs. The games industry is conservative and like all mature industries, full of group think.

Perforce is stable, secure and scales well. If you want to use it, it’s great. It’s expensive though. It broadly uses the same algorithms as Subversion to store data but it’s primary advantage is Streams, tight integration in to Unreal Editor and it scales to HUGE depot sizes.

Unlike GitHub, GitLab or Gitea, Perforce is a stable product over time. You won’t be upgrading every week with security patches you won’t have fancy websites to worry about being hacked, you will basically, every so often, load the admin interface to check why it’s starting to crawl and delete users because you’re running out of seat licenses and the CFO wonders why your sending $$$ to a company in SW England.

Good and bad of Subversion

I’m just gonna add a summary here because you’ve already read a lot.

+ server is simple to setup on Windows, Linux, *BSD or macOS
+ easy to maintain
+ works just as well if not better than Perforce with Unreal Editor
+ is substantially faster than Git.
+ Is supported by UnrealEd out of the box
+ Repo size scales better than Git (but not as well as perforce).
+ Team size scales better than Git (but not as well as perforce).
+ Simple to backup… remember to do this!
+ Free to use.
+ Highly scriptable, much better than Perforce, same as git
+ Integrates with Jenkins out of the box
+ Supports locking by default… and it works and you can cancel/override locks.
+ Subversion is extremely mature. It changes slowly letting you get on with you actual job of making a game.
+ BP “diffing against last depot” is fast because the “depot” reversion is stored locally (see negative [1])
- Tortoise Subversion is the best “desktop UI” for Subversion. It’s worse then P4Win, GitFork, GitExtension, GitHub Desktop. But you’ll mostly be in Unreal or Rider both of which support Subversion very well.
- When you checkout an Subversion repository you get two copies of nearly everything (by design this is to make reverts/diffs fast) and this is why for AAA it’s not good enough. Still better than a version of everything through all time like git-lfs though. [1]
- You don’t get all the bells and whistles of bug tracking, CI, Pull requests that you get with GitHub/Lab and for $$$ with Perforce.
- Perforce streams are a good way to lower bandwidth for remote teams. With Subversion you can (ab)use it’s externals property to create something similar but it’s not as powerful. It does work though. I’ve used externals to create “views” of the repo for design, art and code.
- Perforce support exists. It’s not that good but it is there. Subversion support is the internet.
- Perforce is industry standard and by that I mean people who know nothing about source control will complain you are not using Perforce only to complain just as much if you do use Perforce. Go figure.

Setting up Subversion

This isn’t a tutorial, there are already lots of those out there for your OS of choice

A fresh install of any operating systems will do. Install Subversion, Apache and Python 3 on these machines which are all mature well used packages on most operating systems. There are detailed docs for all platforms on installing Subversion. I’ll just give my set up here. If you must use docker… then there are docker files out there.

You’ll need at least two (metal or virtual) machines if you want a live backup (which I recommend). The backup can be a small PC in your home. You don’t need to give AWS/MS more money. Just make sure you do back shit up!

Here is my (FreeBSD) apache conf file for svn

/usr/local/etc/apache24/Include/svn.conf.

LoadModule dav_svn_module     libexec/apache24/mod_dav_svn.so
LoadModule authz_svn_module   libexec/apache24/mod_authz_svn.so

KeepAlive on
MaxKeepAliveRequests 1000
Timeout 12000
KeepAliveTimeout 15

<IfModule dav_svn_module>
# Add some over the wire compression
SVNCompressionLevel 5

# cache size in KB
SVNInMemoryCacheSize 16384
SVNCacheTextDeltas On
SVNCacheFullTexts On

</IfModule>

<VirtualHost *>
    ServerName XXX_YOUR_SERVER_NAME_HERE_XXX

    <Location />
        <ifModule mime_module>
            RemoveEncoding  .gz .tgz .Z
            RemoveType      .gz .tgz .Z
        </ifModule>

        DAV svn
        SVNParentPath /var/svn/

        # Authentication: Digest
        AuthName "SVN"
        AuthType Digest
        AuthDigestProvider file
        AuthUserFile /usr/local/etc/apache24/svn-auth.htdigest

        # Authorization: Authenticated users only
        Require valid-user
        AuthzSVNAccessFile /usr/local/etc/apache24/svn-authzaccess.conf
        Satisfy Any

        # Allow repo roots to be browsed.
        SVNListParentPath on

        # Do not limit large update requests
        LimitXMLRequestBody 0

    </Location>
</VirtualHost>

There are some tweakables in here but this is what works well for me. Again, there are lots of tutorials about setting up apache+svn.

Setting user password

Apache will be doing user authentication. The above file tells apache where the user password database is. This command adds a user and sets their password.

1	`htdigest /usr/local/etc/apache24/svn-auth.htdigest "SVN" <USERNAME>`

To delete a user, edit the svn-auth-htdigest file and delete the line with the username in it.

If you want groups for finer access control look up svn conf/authz files, you can set groups of users, set access controls to specific paths, set who can read and who can write etc.

Security

I always run access over VPN software. I use wireguard but use whatever you want. This gives me peace of mind. While apache and subversion are very mature and secure it pays to be careful.

If your team is on a LAN then you don’t need a VPN as you can probably trust your team members (ymmv).

Maintenance script

Server shell script to repack/clean repositories overnight. I’m not actually sure if this is needed but it’s mentioned in the docs so I do it. I went years without doing this and never had a problem.

Again this is a FreeBSD csh script.

#!/bin/csh

set SVNADMIN=/usr/local/bin/svnadmin
set ROOT=/var/svn/

set dirs=`find ${ROOT} -type d -depth 1`

foreach dir (${dirs})
        echo "Cleaning transactions ${dir}"
        ${SVNADMIN} rmtxns ${dir} `${SVNADMIN} lstxns ${dir}`
        echo "Packing ${dir}"
        ${SVNADMIN} pack ${dir}
end

Backing up

Subversion has a concept of syncing two repositories. The easiest way to backup your live repo is to have another machine, somewhere in the universe, that syncs all commit from the primary repo to the backup repo. I run this script nightly but you can do it every hour if you want.

Note in 15 years of using Subversion I’ve had one corruption (due to a disk failure) and I had the repo back online in an a few hours. This was basically the time it took to move the data across the internet.

There are many good tutorials on setting this up.

Integration with other tools.

Subversion has various hooks that get called when things happen. The most useful is the post-commit hook to trigger Jenkins and to push a message to discord of a change.

In the trunk of each repository I have a folder called .hooks. When subversion has committed a file to the repo successfully this hooks/post-commit script is called

#!/bin/sh

REPOS="$1"
REV="$2"

CWD=`dirname "$0"`
PYTHON=/usr/local/bin/python3.11

jenkins_username=buildbot
jenkins_secret=<SECRET>

discord_secret=<SECRET>
discord_webhookid=<SECRET>

PATH=$PATH:/usr/local/bin

# create a temp file for the post-commit.py script
tmpfile=$(mktemp /tmp/post-commit.XXXXXXXX)
svnlook cat $CWD/../ /trunk/.hooks/post-commit.py > $tmpfile
if [ $? -gt 0 ]
then
        echo "No post-commit"
        exit 0
fi


export jenkins_username
export jenkins_secret
export discord_secret
export discord_webhookid
export PATH

$PYTHON $tmpfile $REPOS $REV

rm $tmpfile

The script never really changes and stores the secrets I don’t want in the repository (password, tokens etc).

The script copies a python file from the repo called post-commit.py and runs that. This allows each repo to have scriptable actions that are easily changed and the changes are recorded in the repo.

post-commit.py

This file does all the work. It copies a json file from the repository in /trunk/.hooks/post-commit.json and uses data in that to decide what actions to do depending on what files/paths changed.

If you’re going to use this you’ll need to install some python packages

post-commit.json

This is the file contains configuration info for jobs that will be run if certain conditions are met.

Hope this helps someone.

Changes:

240827 - added video on git+lfs not being great.

EOL

MSVC Print All Predefined #Defines

2022-09-24T00:00:00+01:00

Sometimes you really need to know what defines your compiler has on by default. I had no idea how to do this with Visual Studio so I dug in. Here is how to do it.

Open up a VS command prompt, pick the appropriate prompt x86, x64 etc.

enter the following:

> echo int main(){} > test.cpp
> cl.exe /PD /Zc:preprocessor test.cpp

You should see output similar to the following:

test.cpp
#define _CPPRTTI 1
#define __cpp_ref_qualifiers 200710L
#define __cpp_init_captures 201304L
#define __STDCPP_DEFAULT_NEW_ALIGNMENT__ 8u
#define __cpp_rtti 199711L
#define __cpp_decltype_auto 201304L
#define _MSC_EXTENSIONS 1
#define _MSVC_LANG 201402L
#define __cpp_binary_literals 201304L
#define __cpp_constexpr 201304L
#define _MSC_BUILD 0
#define __cpp_attributes 200809L
#define __cpp_inheriting_constructors 200802L
#define __cpp_generic_lambdas 201304L
#define __cpp_variable_templates 201304L
#define __cpp_nsdmi 200809L
#define _NATIVE_WCHAR_T_DEFINED 1
#define __BOOL_DEFINED 1
#define __cpp_sized_deallocation 201309L
#define __cpp_threadsafe_static_init 200806L
#define _HAS_CHAR16_T_LANGUAGE_SUPPORT 1
#define __cpp_initializer_lists 200806L
#define _IS_ASSIGNABLE_NOCHECK_SUPPORTED 1
#define __cpp_static_assert 200410L
#define __cpp_return_type_deduction 201304L
#define __cpp_enumerator_attributes 201411L
#define __cpp_decltype 200707L
#define _M_IX86_FP 2
#define _CONSTEXPR_CHAR_TRAITS_SUPPORTED 1
#define _MT 1
#define _WCHAR_T_DEFINED 1
#define _MSVC_WARNING_LEVEL 1L
#define __cpp_alias_templates 200704L
#define __cplusplus 199711L
#define __cpp_lambdas 200907L
#define __STDC_HOSTED__ 1
#define __cpp_user_defined_literals 200809L
#define __cpp_rvalue_references 200610L
#define _NATIVE_NULLPTR_SUPPORTED 1
#define __cpp_namespace_attributes 201411L
#define __STDCPP_THREADS__ 1
#define _MSC_FULL_VER 193331629
#define _MSC_VER 1933
#define _MSVC_TRADITIONAL 0
#define __cpp_range_based_for 200907L
#define _MSVC_CONSTEXPR_ATTRIBUTE 1
#define __cpp_raw_strings 200710L
#define _M_IX86 600
#define __cpp_unicode_literals 200710L
#define __cpp_unicode_characters 200704L
#define _MSVC_EXECUTION_CHARACTER_SET 1252
#define _INTEGRAL_MAX_BITS 64
#define __cpp_aggregate_nsdmi 201304L
#define _WIN32 1
#define _CRT_USE_BUILTIN_OFFSETOF 1
#define __cpp_variadic_templates 200704L
#define __cpp_delegating_constructors 200604L
Microsoft (R) Incremental Linker Version 14.33.31629.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:test.exe 
test.obj

I turned this in to a simple batchfile

Hope this helps.

EOL.

Unreal: Am I On The Server

2022-09-24T00:00:00+01:00

I’ve been working on an Unreal game for the past few months. One of the things that the whole team has found very frustrating is knowing if the code we are writing is running on the server or the client.

Lots of code only needs to run on one or the other and some code needs to run on both. Unreal has a powerful and slightly confusing networking model combined with numerous ways to run the game from within the editor.

To help me understand the different modes I created the following table. I wrote a small bit of code in my GameMode class to extract some values. I added a break point just after this code so that I could record the values.

UWorld* World = GetWorld();
ENetMode NetMode = GetNetMode();
ENetRole SystemRole = GetLocalRole();
bool bHasAuthority = HasAuthority();
auto GameMode = World->GetAuthGameMode();

PlayMode is the mode you select in the editor (in the Play button drop down).

# is the number of players selected in editor

BP is the number of times my break point is hit. So for two players I would expect to hit it twice (note that this isn’t true in one case).

Has Game Mode technically the game mode only ever exists in the server.

Play Mode	#	BP	Net Mode	System Role	Has Game Mode	Has Authority
StandAlone	1	1	NM_StandAlone	Role_Authority	1	1
StandAlone	2	1	NM_StandAlone	Role_Authority	1	1
		2	NM_StandAlone	Role_Authority	1	1
Listen Server	1	1	NM_ListenServer	Role_Authority	1	1
Listen Server	2	1	NM_ListenServer	Role_Authority	1	1
		2	NM_Client	Role_SimulatedProxy	0	0
Client	1	1	NM_DedicatedServer	Role_Authority	1	1
	1	2	NM_Client	Role_SimulatedProxy	0	0
Client	2	1	NM_DedicatedServer	Role_Authority	1	1
		2	NM_Client	Role_SimulatedProxy	0	0
		3	NM_Client	Role_SimulatedProxy	0	0

It all makes sense to me apart from in StandAlone mode. In all other scenarios NetMode and SystemRole have reasonable values. You can easily say “this code is running on the server” when NetMode == NM_ListenServer or NetMode == NM_DedicatedServer.

So what is going on in Stand Alone mode? In standalone mode everything is running inside one process, the same process the editor is in. So The code is both the server and the client. This makes some things easier to debug and other things just plain confusing. I think this is the scenario for a split screen game, so it’s important your code works in this mode as well.

As a side note Visual Studio has a plugin to enable child process debugging. With this plugin you can launch the editor from VS, play the game and launch child clients and the debugger will attach to them and allow you to set break points and view memory, very useful.

Hope this helps.

EOL.

OpnSense Wiregaurd VPN Road warrior setup.

2022-07-22T00:00:00+01:00

Table of contents:

Enable Wireguard on OpnSense
Create a local wireguard instance/configuration
Create a client end point
Add OpnSense Fireguard rule
Client Configuration
What is missing?

I found getting WireGuard to work on OpnSense a “little tricky” to say the least (but far easier than OpenVPN).

The setup I wanted is the classic Road Warrior setup. While away from my office I need my laptop to have access to the office LAN, so I can remote desktop to my work PC and I need all my mobile phone data (iOS) to be routed securely via the office.

I started with the official docs for a Road Warrior Setup. This didn’t work out of the box for me. So I’m detailing my setup just incase it is useful for others.

Enable Wireguard on OpnSense

Note: you will need to install the plugin first!

Create a local wireguard instance/configuration

The public and private key can be generated for you. Remember to keep the private key private!. Set a high listen port (51820 or higher).

I decided to go with the private IP range 10.10.10.x for my VPN. So the server (tunnel address) is on 10.10.10.1 and all clients will be on 10.10.10.2, 10.10.10.3, …

Create a client end point

Once you’ve created an end point remember to enable it in the Peers options in the local configuration.

Add OpnSense Fireguard rule

Gotta let those packets in through the WAN. Make sure that the port is the same as the one you set in the WireGuard configuration.

I think WireGuard created rules for me allowing packets to go from the WireGuard interface to any other group. For example from WireGuard (10.10.10.x) to LAN (192.168.1.x).

Client Configuration

note: endpoint is the IP:PORT pair of the WireGuard router.

I installed the official Windows client. Most of the values in here are just copies of the ones from OpnSense. Crucially you will need to set the private key that you generated on the server in here. I would show you the editable configuration (by clicking the edit button) but when I did that WireGuard stopped responding and now after restarting it the configuration has been deleted! I mostly followed this tutorial to get Windows set up.

The bit in the client configuration that was NOT intuitive for me was the Allowed IPs. The documentation seemed to imply that setting 0.0.0.0/24 allowed the client to route to any network. This didn’t work for me at all, and I had to explicitly allow 10.10.10.0/24 for access to the DNS server in this subnet and 192.168.1.0/24 to allow access to the LAN as well as 0.0.0.0/0 which allowed my clients to access networks outside my LAN but still via the VPN.

The settings for iOS were similar. I installed the official WireGuard app from the App Store.

What is missing?

For me, the only thing that is missing is some form of two-factor authentication. This isn’t a deal-breaker as you need to log in to my laptop, and you need my face to unlock my phone. However, I feel it would be good especially for my laptop for some form of added authentication to take place before connecting to my VPN.

Hope this helps someone else.

dazza.

Adding “Fork Me on Github” to Pelican Templates

2020-04-24T00:00:00+01:00

I just wrote a blog about a project I uploaded to GitHub and I wanted to have a “Fork Me On GitHub ribbon” in the top right of the page. This is the cool thing to do nowadays after all.

I write my blog using a static website generator called Pelican. One of the advantages of Pelican is that the templates used to generate the static site are easily modifiable. Here is what I did.

Firstly I found someone who had done the dirty work of creating a CSS ribbon, Thank you Simon. For Simon’s ribbon to appear I needed to add one CSS file and an anchor to any page with a GitHub link in it.

Adding metadata to articles in Pelican is very easy. For any article that needed a GitHub ribbon I add some metadata named githuburl. Here is the meta data for my article:

Title: CSGJS-CPP on GitHub
Date: 20200424
Category:
Tags: GameDev, C++
Authors: dazza
githuburl: https://github.com/executionunit/csgjs-cpp

So now when processing an article and generating HTML the article object will have a member called githuburl.

The blogposts are rendered using the article.html template. I added

{% block extracss %}
    {% if article.githuburl %}
        <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/github-fork-ribbon-css/0.2.3/gh-fork-ribbon.min.css" />
    {% endif %}
{% endblock %}

to the top and at the bottom.

{% block endbody%}
    {% if article.githuburl %}
        <a class="github-fork-ribbon" href="{{ article.githuburl }}" data-ribbon="Fork me on GitHub" title="Fork me on GitHub">Fork me on GitHub</a>
    {% endif %}
{% endblock %}

This code creates two new blocks and if article.githuburl has a value the extracss block will contain the CSS link and the endbody block will contain the link to the GitHub page.

To make these blocks rendering in to the output HTML I need to tell the template engine where to put them. So I modified base.html which all templates inherit from.

<head>
    ...
    {%block extracss %}{% endblock %}
</head>

<body>
    ...
    {%block endbody %}{% endblock %}
</body>

After that I rebuilt my site and now I can have a little GitHub ribbon on any page just by adding some meta data to an article. Pretty cool.

You can see the article here complete with a lovely GitHub ribbon.

I hope that makes sense.

dazza.

CSGJS-CPP on GitHub

2020-04-24T00:00:00+01:00

I spent some time playing around with CSGJS over the weekend. It’s a great library. I thought it might be interesting to port it to C++.

After quite a lot of googling I found that someone had already done most of the hard work here which was nice and they had kept the MIT license which was even better.

I decided I’d take their work and see if I could improve it a little and learn how it worked.

The fundamental algorithm used is to take two sets of convex polygons that make up a 3D model, create separate BSP trees and then perform union, subtract or difference algorithms upon the target BSP tree. The algorithm is actually very nice and well documented here Constructive Solid Geometry Using BSP Tree - Christian Segura1, Taylor Stine2, Jackie Yang.

The code I took worked pretty well but I managed to optimize making it about 40% faster and removed the memory leaks.

Here are some examples of the output.

A gourd with a cylinder subtracted

The same gourd with the cylinder added (union)

Cube with a sphere union and cylinder subtracted along x, y and z axis

The good thing about the algorithm is that it’s recursive nature and simple testable stages make it pretty easy to implement and test. The downside is that the creation of BSP trees both at the initial “list of polygons to BSP” stage and the processing stage cause a lot of allocations making this unsuitable for real time operations.

Anyway, all my changes and test harness are available on git hub for people to play with.

Smith and Winston - Update #11 - Halloween

2019-10-28T00:00:00+00:00

It’s our first sale! Halloween isn’t really a British thing but you’ve got to go with the flow.

So if you’re looking for a bargain this All Hallows’ Eve look no further than Smith and Winston.

We’ve added some spooky helmets for you to play the game with as well! Everyone gets them for free. Just look in your Collectables menu and set your head!

Have fun exploring and destroying.

dazza

Develop: Brighton 2019 Indie Showcase

2019-06-18T00:00:00+01:00

We are pretty chuffed to announce that Smith and Winston has been selected for the Indie Game Showcase at Develop: Brighton 2019 here in the UK.

Develop is an industry event where developers from all over Europe get together and talk shop. We’re very surprised and quite elated that Smith and Winston is one of the 10 games that has been selected by an expert panel. We didn’t even have to bribe them with cake!

With all ten games being of such a high standard it looks like it’s gonna be a tough competition.

dazza

Smith and Winston - Update #10 - Power Cubes, Shippy, New Levels, More Debris and much more!

2019-05-23T00:00:00+01:00

Update #10 is up on Steam and itch.io NOW! Don’t delay, buy it or wish list it… what’s the worst that could happen!

New levels to explore

Spynall Tapp

Where the demons dwell if you can find the secret door at least.

Asteroid

You’ve made it to the surface of the Asteroid. The space ship that the VOID used to gather the monoliths to unleash his power… it’s a bit weird on the giant cube.

These are the final two levels before you enter the Asteroid to face the VOID itself.

Escape from the teacher

Listen, we all know that you hate training levels. But we also know that some people need them. So we’ve added a get out quick door at the beginning. So if you are super l337 hard core haX0r twin stick aficionado then you don’t need to sit through it.

It’s certain flowers that give you coins.

We got an interesting bug report that sometime you picked up a power cube but it didn’t appear in the HUD. After some digging we realised that it wasn’t clear that some flowers give you a coin. So now when you get a power cube an icon appears to let you know it’s a power cube you’ve collected and not a coin.

Power Cubes

Speaking of power cubes. You collect them to repair shippy, well at least that’s what she tells you at the beginning of the game. Well now they are needed to power the spinning portal doors. After feedback from EGX we realised that people were just powering through the levels and not taking the time to smell the roses and explore a little. There is a lot to find in all the levels so this is a little hint to chill a bit, destroy a bit more and dig for some POWER!

The number of power cubes needed is written at the top of the door because the aliens that inhabit this shattered ring world are all highly organised and probably have a spreadsheet about these sorts of things.

Sarcastic Shippy

When you jumped to a new level Shippy told you some useful information about how many enemies you’d killed, how many power cubes you got and all that stuff. Well that was far too nice of Shippy so now she tells you sometimes useful information and sometimes sarcastic information.

Bug Fixes and tweaks.

Screen SHAKE Shenanigans!

Screen shake is worked out for each co-op player independently so now, just as you were making that difficult boost across a gap, your coop partner can’t throw a grenade making the whole screen shake.
If the screen was shaking because, say, a large block was moving, and you threw a grenade… well all hell broke loose and the screen shook like a wet dog. That’s fixed now.
If you’ve turned off the screen shake then maybe give it another go.
Better physics on low end machines
Physics uses more cores if you’ve got them
Up debris in options > game > debris

Faster loading and rendering

We’ve sped up the loading of levels a tiny bit by being a bit more clever… or less dumb. We no longer build the same model thousands of times per level and just do it once. Thank you for recognising my genius.

Have fun exploring and destroying.

dazza

Notes on vr development

2019-05-10T00:00:00+01:00

Prompted by a question on reddit’s gamedev subreddit here is a slightly more fleshed out answer to

How is developing for VR different than a standard 3D dev cycle?

In 01/2018 we worked on a VR prototype of Smith and Winston, ultimately we abandoned it because we couldn’t attract enough investment to take it forward. Here is what we learned, hopefully this is useful to anyone else embarking on a VR project even though this was learned over a year ago.

Interaction

One of the first things you’ll experience in VR is that some types of interaction that are rewarding in a 3D or 2D game are so different that they are completely useless or make you vomit.

Things that are annoying like “Press triangle to continue” are a PITA in the real world and if you can’t see the controller it’s impossible to remember where triangle is anyway. The solution of placing the controller in the scene, something we did for Smith and Winston demo, doesn’t work in all games. Astrobot does this really well by making the controller an integral part of the players experience.

Putting the controller in the scene helps but not always

The sheer act of looking around in a 3D game is interesting but it can be completely absorbing in a VR game. Even beyond the trivial first contact with VR the act of being in a game is intensely rewarding to the player.

So if you bring a traditional game, like we did with Smith and Winston, you quickly find a lot of what you’ve learned, implemented and become accustomed to is no longer useful.

Sound Design

Sound design is the ugly step child in game development. Everyone knows it’s there but no one pays enough attention to it. In VR the minimum requirements are much much higher. I’d go as far as to say potentially two to four times more costly for an indie. For the AAA guys, they are already doing full surround sound 7.1 Dolby mega THX tastic stuff that people play out of their crappy PC speakers. For them, the audience may actually hear their lovely work. For indies, you need to step up your game.

In VR, because the players head is IN the game (and thus their ears are in the game) and because nearly all VR is played with headphones, the potential for sound design is much greater but also what is needed is much greater. The player can pinpoint exactly where the sound is coming from relative to them and they will look at the sound source.

this image is from the 3D (non vr) version of the game

In Smith and Winston we had a very low pace puzzle level as a break from the destruction. The sound design for this level is a very light wind, the occasional Cicada chirp and a single bird call (that can play from five or so different locations). That is all that is needed. The player gets it, it’s restful.

However in VR the bird call was annoying as you can never see the bird despite being able to locate the invisible sound source pretty well. We’re all used to not being able to locate the Cicada sound but we expect to be able to see the bird.

There is so much potential for immersive sound design in VR but there is also so much that can be subtly wrong that disconnects the player from the scene.

Textures will all need work

The actual displays for each eye are very low resolution and the image you send to the headset is very distorted. This tends to exaggerate any slight movement of pixels and noise within textures. The “small voxel” art style in Smith and Winston naturally creates a very high frequency image and when distorted the middle distance shimmered and looked terrible. We toned all of our colours down to compensate for this. It’s basically was not possible to take all of the levels we’d already designed and just explore them in VR.

The level of detail in the scene dramatically affects how the brain reacts to motion. So games like Rez with limited detail can get away with extreme speed/direction change where as doom can make you spew in 30 seconds. It takes a lot of time to find the sweet spot for your game and art style.

Implicit design.

There are very many things about playing a game that you don’t need to explicitly tell players. Everyone knows how the analog controllers work for example. In VR those tropes haven’t been established yet. As time goes by this will be fixed and stand out titles like Astrobot are setting standards for people to follow. We’re not there yet though.

Captivated

In games like Astrobot, Moss and in our demo players are totally captivated by tiny life forms that they can bend over and look at. It’s a deeply rewarding and weirdly emotional experience for players and it can change your game completely. When the robots in Astrobot look at you it is a very intense experience that cannot be replicated in 3D (nor in movies I would add).

I couldn’t find a picture of looking at Smith or Winston but here is a pick up the player uncovered

Time challenges

The tweak/test cycle is a LOT longer. It takes time to put the headset on/off and you need to rest your eyes/brain a lot more for VR work. It’s easy to work too long and get car-sick.

Budgeting

A lot of the recent indie/small studio boom has been enabled by the off the shelf engines. This has dramatically lowered costs and risk. In VR those gains are partially lost. The increased sound design and slower design/tweak/test cycle increase costs and risk. This new risk/cost profile has to be balanced with the smaller market that VR currently offers. Again, over time this will hopefully change as VR grows. A lot of publishers are not willing to take on the increased risks at the moment so the actual designs they will consider are more limited.

Anyway, hope this helps someone who is researching doing a VR game.

Smith and Winston - Update #9 - Save the Day

2019-04-18T00:00:00+01:00

Save the coop

We’ve turned on save games for coop. So now you can officially continue a coop game you’ve been playing with your buddy.

Coop rebirth

If one of the coop player runs out of lives they will be given a free life with a full shield if you can get to the next level. We’re prototyping an in-level recovery mechanism where a pick up can recover the lost player. We’re not sure if the pickup should come from killing, destruction or just add it to the extra life pickup. Any ideas?

Buddy Mode

We think this is a big one. In buddy mode player 2 can never be killed and if they fall off of the level they get respawned. We wanted a way to play Smith and Winston with our young kids. Hopefully you’ll find it useful. It’s part of our design mantra of always making the player feel powerful.

There are some really interesting tactics that you can use in buddy mode with your kids or less-gamer-like friends and it’s fun and that’s all that really matters in life.

Oh coop partner, wherefore art thou!

You can turn on a little icon that appears when your coop partner goes offscreen. Yeah the levels can get quite confusing after blowing them to pieces and spinning around for a few minutes :)

Language update

We’ve updated all the German, French and Spanish strings in the game with google-translated strings that match the English. They got out of sync with the EGX Rezzed build. Sorry.

No more clever, adjust the slider

We tried to be clever and add a dynamic debris load balancing system and it didn’t work. The code tried to monitor how long the CPU took to calculate all the debris physics and adjust the amount of debris on screen in a super clever way. Well it turns out it was bloody rubbish! So now there is a slider in options > game that you can slide up or down to increase and decrease the amound of debris. It’s stupid, but it works.

Shippy

We’ve added some more sarcastic shippy comments through out the levels because she’s funny and we like adding sarcastic messages when we get a few minutes.

Have fun exploring and destroying.

Smith and Winston - Update #8 - Cooperation for all

2019-04-10T00:00:00+01:00

Shattering

For a long time we’ve had the world blowing up in to tiny voxels. This has been very satisfying for everyone. We’ve made it even more satisfying now by creating big chunks of floor and walls from the world when you blow it up.

Tree Collapse

Cutting the bottom out of a tree is one of the first things that players do. When it doesn’t fall a lot of players feel cheated. So now the tree will fall down shattering in to bits in the process. This is oh.. so… good :)

Head over Heels for Coop.

We recently showed Smith and Winston at EGX Rezzed in London. Coop mode went down a complete storm. “Best Game of the Show”, “So satisfying” were just two of the many comments we got from all different types of gamers and we’re still beaming from hearing all the kind words.

Coop mode isn’t complete and there are still a few issues but it’s so much fun we want everyone to join in with a friend and blow some stuff up.

One notable ommision is that we haven’t got save games working and when a player dies there is no way to revive them just yet but fixes for those are coming soon.

Dig in.

Better Debris

One of the challenges with Smith and Winston is managing all the tiny cubes that fly around. They are relatively expensive to draw (for what they are) and they are very expensive to do the collision for but oh they are so satisfying.

So in this update we’ve added a more flexible system where if they are getting too expensive we remove them from the scene more quickly keeping the framerate up for all players and allowing players with monster machines to get even more debris!

This is especially useful in coop where people seem to spend most of their time mining out the floor from under their ‘coop partner’ :)

Mining freeze

At the EGX rezzed show we learned that a lot of users were just using the mining laser to shoot everything, the world and the enemies. This is cool but when someone said “I don’t think you should give the player a shotgun from the beginning!” we realised that we’d got the mining laser all wrong.

So for mining the laser hasn’t changed at all but now it freezes enemies it hits for a second or so giving the player time to change weapon. Give it a go and see what you think.

Quicker level restores

With all this new destruction we started to notice that when the player died and we have to revert the level back to it’s original state it was taking too long. This has been even more optimized now so it doesn’t get in the way of your fun.

New title page and new intro movie

No one likes a long boring intro movie but we’ve made one just to annoy you even more. Trust me it’s very short, gives you an insight in to Shippy and why she’s a little mad and hopefully as we finish up the single player mode you’ll enjoy her more as a character. We’re working really hard to making her a richer and more rewarding part of the game.

We changed the title page just because it so cool to see Shippy in space!

Thanks to all our supporters

Lastly but definitely not leastly we’d like to say thanks to all that have bought the game and supported us. We are just two guys making a game. It’s completely self funded, we don’t have a wealthy backer or publisher and sometimes we need to take on contract work to pay the bills. It’s really frustrating but that’s life. Once again, thanks for your patience and kindness.

Tweaks

A lot of small things that players have asked for:

Disable ALT-F4 from killing the game.
Better controller detection and actions when unplugged/plugged in
Correct controller names in the HUD
Enemy shots are more controlled, sometimes they fired at weird angles which sucked
Turn on/off Depth of Field (video menu)
Turn on/off Shadows (video menu)
Turn on/off Bloom (video menu)
Enable/Disable screen saver kicking in (video menu)

Thanks for reading all this, I’d forgotten we’d done half of this stuff. The next update hopefully won’t take so long to get out.

Have fun exploring and destroying.

Smith and Winston - Play Smith and Winston CoOp at Rezzed 2019

2019-03-30T00:00:00+00:00

If you’re coming along to Rezzed 2019 then stop by our booth and give Smith and Winston a play and chat to us. Both Charlie and I will be there in our exclusive Smith and Winston tshirts so we won’t be hard to miss.

Exclusive Coop

Coop is still a bit buggy but we’re enabling it for this special event. As we’ll be there we’ll be able to tell you what’s gone wrong and apologise :)

Exclusive Free Voxel

We’re giving away exclusive voxels that have travelled from the far side of the asteroid. In our dimension they appear to be flattened but can be folded and glued back in to “three” dimensions for us to view ;)

Boob shot!

Charlie and I will be wearing these exlusive tshirts so we are easy to find. We’re pretty happy how they turned out in the end. Our names are across our bums as our names and big like our bums.

A preview of our stand

The booth will also have a massive Smith and Winston logo over it as well so… yeah you won’t be able to miss it.

Hope to see you there. There are lots of indies and mainstream games to play at Rezzed… but the indies are the best!

Smith and Winston - Update #6 - New Levels, Shield is Live and a Rejig.

2019-03-06T00:00:00+00:00

Story and Arcade Merged

If you’ve been following along you’ll know that originally there were two game modes: Story and Arcade. With the introduction of the shield (see below) and thus the removal of one shot death these two modes seems a waste of time. So they are merged. The downside of this is that everyone’s saves games are now invalid so, unfortunately, everyone has been kicked back to boot camp and has to start the mission again. The good news is that the mission is much clearer, much more balanced and starts a lot easier because of the shield.

Introductory Offer

Lets be honest no one really likes a cut scene but they are sometimes necessary. We found that people were asking questions like:

Who are Smith and Winston?
Am I playing as Smith or Winston?
Why is shippy such an asshat?
Is shippy the ship?
How many potatoes are there is each voxel?

So when you start a new game you get to see a little 30 second intro sequence which helps to set the scene a little.

Training

The cut scene plops you in to a little training area now where you can learn the controls. We’ve removed most of the introduction Shippy Beacons from the start of the game now. A lot of players just ignored them and Shippy’s voice was just too annoying

Calm down Shippy!

We’ve re-synthesized Shippy so she’s FAR less annoying to the players ears, especially when wearing headphones. So if you’ve turned down the shippy volume… you may want to turn it back up for a bit.

Shiela Sha-poo-pee Shields

We’ve added a shield. This means you can take three hits before losing a life. This makes a huge difference to the game. It’s amazing. We said we’d never do it but in the end, the players were correct, the game is better with a shield.

Bossing the Boss around

Based on the feedback we got about the Chopper Boss we’ve moved him to be the first boss that you see. It’s a better boss, it plays better than bunker boss and this gives us a chance to revisit the Bunker Boss and make him better.

New Levels

We’ve added three new levels to the game… each with distinct eeriness, distinct enemies and one of them is based on my personal addiction to Paprika Pringles.. but it’s probably not clear which.

Anyway, that’s it for now. We’re going to get a build ready for EGX Rezzed in London 4th-6th of April, 2019. If you’re around be sure to drop by and say hi.

Have fun exploring and destroying.

be seeing you

dazza.

Smith and Winston - Update #5 - Holodeck Lessons

2019-02-22T00:00:00+00:00

Another quick update with some bug fixes and a new BETA area for your feedback.

Hollow Holo

We hear you and agree we’re really not happy with the way we use Shippy to teach the player how to play Smith and Winston. It’s boring and annoying. Training areas in games are always hard tbh so we’re having another go at it.

As part of a brand new intro sequence the player will be sent, by shippy, to a holodeck training area to learn the buttons and get a feel for controlling Smith and Winston before getting stuck in.

The shippy text is all provisional and untranslated and we’ve not added Shippy’s patented rudeness or voice yet

What do you think? Is it a better way to introduce the game? Please give feed back in the discussion area on Steam or itch.io. More feedback on the Chopper boss is very much appreciated as well.

After this the player will see a little cut scene explaining how the ship crashes and then they are straight in to playing the “second” level “Sunshine and Showers”

Bug Fixes

Toilet Sound

If you happened to die in a bit of the game with reverb turned on (echoy sound effects etc.) then you were stuck playing the rest of the game like you were in a toilet. Should be fixed now.

White…. so white

Some textures didn’t render correctly in OpenGL on any platform. This was because we were blending two texture together one of which didn’t exist… yeah, that’s fixed now as well… oops.

Less chatter, more batter

That’s it, have a great weekend, Explore and Destroy and if you’ve got a bit of spare time jump in the discussion area of Steam or itch and tell us what you think of the holodeck, chopper boss or anything else about the game.

be seeing you

dazza.

Smith and Winston - Update #4

2019-02-17T00:00:00+00:00

We’ve released a new update to Smith and Wintston! This is mainly a metric ton of tweaks to the first half of the game to make it more balanced and more fun to play. Thanks to everyone that gave us feedback and helped make the game even better.

Features:

Added in the Chopper boss. This is the middle boss in the game. He’s pretty epic (testable via the new BETA TEST menu)
Added a BETA TEST menu so player can test specific features of the game and give us feedback without destroying their Arcade Save games. We’ll be adding things to this menu and asking for feedback a lot we think.
Added a button for twitter and facebook in the title screen
Tweaking/Balancing of lives and ammo based on player feedback and streamer videos
Added disabled story and coop buttons to the main menu so players know they will be there activated at some point :)
After the player dies in a level specific enemies don’t restore so it becomes ‘slightly’ easier to finish the level. The cost to the player is that they will get a slightly lower score because there are less enemies to kill.

Beta Test

You can jump straight to the Chopper Boss from the new Beta Test menu option. You can test the boss and give us feedback without affecting your Arcade Save game. We’d love to hear from all players ːsteamhappyː

Engine Changes:

Upgraded BGFX rendering code. At least 10% faster on all platforms.
Added Metal rendering engine for macOS (8x faster than OpenGL)
Added OpenGL rendering engine for Windows
macOS and Windows can select the rendering engine to use in Options > Video

Bug Fixes:

Fix for some of the trophies not being present
Skip cutscene text is now correct for controller, KB/Mouse users.
Ping Game Boy helmet renamed to the correct Pink Game Boi
Chopper based enemies can now be hit by grenades
Mining Laser now shoots from closer to the player so that when right up against the wall it actually hits it. This makes running through the caves and mines a lot easier.
In some levels the player plows through snow or mud, after they died this stopped working, this is fixed.
Some GameObject (enemies, pickups etc) were not restoring after player death
long german and spanish phrases should now in to text boxes
custom in game cursors appear in front of the shippy text.

Once again, thanks to everyone that’s played the game and given us bug reports and feedbacl. It’s really means the world to us so jump in to the forum and tell us what you think!

Smith and Winston - Update #3 Patch 2

2019-01-22T00:00:00+00:00

Today we released a 32MB patch to Smith and Winston on itch.io and Steam fixing a few small issues, adding a few little things and tweaking the game play.

New Features:

Camera shake can be adjusted in Options > Game > Camera Shake
Controller rumble can be adjusted in Options > Game > Rumble
Added three custom mouse cursors for KB/Mouse players, see Options > Controls > Mouse.
Added a languages menu on Option where you can select the temporary translations for Spanish, German and French or the default English.

Tweaks to the game:

Added a glow to in game switches to make them a bit more obvious.
Added a flashing light to Shippy beacons so you can ignore them even more :)
Lava wasps now get killed by the exploding Lava eggs.
Spiders can be killed with grenades.
Dashing makes you invulnerable to explosions as well as bullets (while you are blue). So you can dash through spiders.
Some more sound design done in the Swamp of the Reanimated.
Added in more collectable helmets ( they have all now been placed in the game though some of them are on levels not yet released).
Balanced pick up and weapon distribution e.g all main levels should have at least 2 lives in them.
Added some more air pick ups, the number of these dropped off quite quickly as you progressed through the levels, there should now be a few more to test your dexterity.
Small voxel changes to the levels, e.g. fixing voxels that the player was getting caught up on.
Fixed lighting issues in Swamp of the Reanimated. Happened when the player went back through some of the portals.
Made some of the hidden portals a little easier to spot.
Labyrinth secret level is now a lot more forgiving and a wrong portal returns you to the same part of the level rather than the previous bit. Also the portal to the level exit has been differentiated from the other ones.

Bug Fixes:

more robust loading of corrupt save games.
no longer hide the cursor when playing with a Controller. This leads to annoying issues for users when like to use the mouse to tweak options with the mouse but play the game with a Controller.
When using the menus, pressing back selects the menu you’ve just come from to make menu navigation more clear.
Menu sliders now scale better for small resolutions (1024x768).
When setting a controller action to use the ‘B’/Circle button (which is also menu back) set the option but do not go back to the previous menu.
KB/Mouse shooting is now more accurate, the players gun now faces the mouse position not the player.
Fix for PSX controller Circle not appearing in Shippy dialog boxes
Linux, using a non-English locale no longer stops levels loading and doesn’t create corrupt userprefs files (thanks Stephan) *

Known Issues:

long German and Spanish phrases don’t fit in to text boxes.
custom in game cursors appear behind the shippy text.

Happy Exploring and Destroying

dazza

itch.io Desktop App on Fedora 29

2019-01-19T00:00:00+00:00

I just tried to get the itch.io desktop app running on a fresh VM of Fedora 29. I used the Fedora-Workstation-x86-64-29-1.2.iso and then ran an update to get a few patches.

Unfortunately when I downloaded the excellent itch.io app from itch.io and installed it

chmod +x ~/Download/itch-setup && ~/Download/itch-setup

it then just refused to run… as in nothing happened.

Digging a little further itch installs itself in to

1	`.itch/app-24.5.0/itch`

NOTE: the version number is likely to change as itch updates it’s app very regularly.

running from the command line it became clear what was wrong:

.itch/app-25.4.0/itch: error while loading shared libraries: libXss.so.1: cannot open shared object file: No such file or directory

So libXss is missing, should be easy to install, I wonder what it is? X Screen Saver… oh, OK!

sudo yum install Xss

fixes the issue and now you can download and run Smith and Winston

Hope that helps someone out there!

Smith and Winston - Update #3 Patch 1

2019-01-14T00:00:00+00:00

Today we released a 4.3MB patch to Smith and Winston on itch.io and Steam fixing a few small issues that have been reported by players and one we noticed in a Twitch steam.

Hot low end action

On lower end machines players are having to turn down the amount of debris. THIS IS NOT ACCEPTABLE. So we’ve tweaked the physics a bit to get some more juice from your CPU and get some more debris on screen at 60 fps.

Keeping your head!

Because we are idiots we didn’t think about heads properly. So if you collected one of the hidden Collectable heads in the game you could never switch back to a standard Smith or Winston head. Well now you can.

More text… or space for it

Some of the shippy text when you’re at the crash site didn’t fit in the little shippy box. We know no one is going to read this text but we really want you to be able to ignore all of it properly, in it’s entirety with aplomb.

No dead flying!

When you were lying flat on your back, shot, wondering why you even started exploring this cursed ring world of DOOM you could press the boost button and fly up. This wasn’t very useful to you and in many ways it was quite funny but NO… STOP IT…. you’re dead ;)

No more equality

If you pressed the equals key (‘=’) you seemed to randomly warp around the level… even to places that you can’t get to normally. We use this debug feature to quickly test new areas of the level and you were never supposed to have such powers because we just can’t trust any of you naughty people can we? Look out for this feature accidentally being re-enabled because “we are idiots” ;).

No more HOME

Another oversight was that pressing the HOME key brought up a very useful debug window with loads of wibbly buttons and sliders and gizmos and yeah, we’re idiots, that’s not for you. Even I don’t know what most of those buttons did.

Glyphs?

A glyph is a fancy word for one of the letters in a font. We cleaned up the glyphs that represent the controller buttons for PSX and XBox controllers so they are less… poopy.

Happy Exploring and Destroying

dazza

GameMite review Smith and Winston Early Access

2019-01-12T00:00:00+00:00

The lovely people over at GameMite have reviewed Smith and Winston. I particlularly liked the ending where they say:

Anyone who loves a good, exciting twin stick shooter should find a lot to enjoy here.

Great to see people covering the game and getting it’s weird mix of metroidly looking aroundy shootyness.

be seeing you,

dazza

Smith and Winston - Steam Early Access

2019-01-09T00:00:00+00:00

We are really pleased to finally be able to say that Smith and Winston is on Steam. We really enjoyed releasing on itch.io first and we think that the smaller community helped us prepare for the much larger and faster moving Steam community.

As well as releasing on Early Access we are starting to ramp up press activity so people know about the game. We’ve opened up the Smith and Winston Press Kit page to everyone. There are loads of lovely images, gifs and a few movies in there as well. Feel free to use them to talk about the game.

New gameplay trailer

Plans for itch

Just because we’re on Steam we don’t plan on dropping support for itch.io. Itch.io is a great site for indies and we plan to support it. We’ll be updating itch with the same code as Steam (excluding things like Steam achievements and cloud saves for obvious reasons).

If you bought the game on itch.io in the past few months you can get a free Steam key via your Purchases section in your itch.io account. Your save game should just work as well so you can fire up your Steam copy and carry on from where you left off. At some point we’ll move to Steam Cloud saves though and your save games will no longer be the same between itch.io and Steam versions.

Plans for Steam

On Steam we plan to add a lot more achievements, cloud saves, leader boards and more controller support.

Plans for the game

We’ve got a long list of things to add to the game including:

13/25 levels are not ready for the public yet (too buggy or incomplete) and are being worked on as we speak.
2/3 bosses have not been completed yet.
Split screen two player coop mode isn’t enabled as it’s too buggy and the levels need balancing.
Story Mode isn’t enabled as the save games don’t reliably work yet.

So there are things to come for itch.io and Steam users.

Have a great new year and we hope you enjoy Exploring and Destroying.

be seeing you,

dazza

How I support Windows, Mac and Linux

2019-01-02T00:00:00+00:00

10th Jan 2019 12:03 GMT - I just wanted to write a positive article that might help others. So please don’t use this post as a stick to beat others. It’s a complex issue of dev experience, team size and frankly money. We’re all just trying to make great games - dazza.

So as usual there is some controversy about supporting Linux (and macOS for that matter) if you’re a game dev. I am a game dev and our current game Smith and Winston support Windows, Linux and macOS. So how do I as a lone coder in a team of two support three platforms and is it all easy and free?

How?

Custom Code

Firstly we use a custom engine and this makes it harder but also better. Here is a blog about how I assemble the engine. I’m sure it is a lot ‘easier’ to just use Unity, Unreal, Godot or any of the other engines out there. The easy path is not necessarily the most fun or rewarding path to travel though.

The upside of custom code is that I basically build upon the best for me, my game and my customers. libSDL2 is fantastic and takes 90% of the pain away from platform support. BGFX is an amazing library for rendering, written by someone who clearly knows about cross platform support and is dedicated to producing quality code.

Custom code gives me the maximum chance of being able to debug a platform specific issue. Not only is the source code available but because I integrated it I know how it works. I hate the idea that I would have to say to a customer that “I can’t fix that issue because there is a bug in Unity/Unreal that I can’t fix and will never really know when it’s going to be fixed”. This is a matter of professional due diligence. Of course there are hidden corners of all the libraries I use so I can’t guarantee anything but I at least have a fighting chance.

From day one I intended to support multiple platforms. I didn’t code for all of them from day one but I knew that where ever I touched the OS I was going to need to think about platform differences. Common areas are files, file paths, rendering, threading, sound, date and time and localisation. You just need to think about it. If you didn’t compile it then you may need to wrap it in a cross platform API.

Machines and VMs

I dedicate hardware to building and testing macOS, Windows and Linux.

I have a machine dedicated to building the engine and game datasets on all platforms in all configurations all the time. It has many VMs running. It just sits there waiting for new code to be added and the opportunity to make me look stupid. It emails me with build errors and warnings. I hate that machine. Here are the configurations it builds:

Windows
Windows + Oculus
Windows + Steam
Windows + Steam + Oculus
Linux
Linux + Steam
macOS
macOS + Steam
FreeBSD
(consoles I can’t talk about)

all of them in debug, release and profiling configurations. So that’s 27 targets I can talk about. While it’s only really four platforms each of the sub-platforms offer opportunities for me to break things. If you make finding these faults easy and fast then you’re much more likely to fix them early and quickly.

This means I am totally relaxed about macOS and Linux as platforms. There are no surprises for me when it comes to testing and deployment. I use a custom python build system that constantly loops through (checkout, cmake/premake/GENie, build, report). It’s only a few thousand lines of code but it’s saved me many hours by catching code that breaks a platform I am not working on.

I also rotate my primary development platform spending time in Windows, macOS and Linux. This means I know the platforms and tools well.

I’m old

I’ve been writing games and engines for 30+ years so none of this is new, I have a lot of experience. But you only get the experience by doing it and not making excuses.

Separation of game and engine logic

Our engine uses Lua for game logic and C++ for the runtime code. This separation means we can focus our testing depending on what has changed between releases. Mostly the engine stays the same so we can focus on testing the actual elements of the game and the few bug fixes the engine gets.

Why?

Moral Argument

There are a lot of good things to say about Microsoft. Monopolies are not good for anyone though. While you can’t break a monopoly on your own you can play your small part. Microsoft is an infinitely better company now than they were 10 years ago because they’ve had a few bloody noses. Let’s hope someone does the same to Google and Facebook pretty soon. So by supporting the minor platforms you are making the major platform better.

Quality Argument

By forcing the game through different compilers (Visual C++, Clang and GCC) you find different code bugs (leave the warnings on max). By forcing the runtime to use different memory allocators, threading libraries and rendering technologies you find different runtime bugs. This makes your code way more stable on every platform. Even if you never deploy your code to a non windows platform just running it on Linux or macOS will expose crashes instantly that rarely appear on Windows. So delivering a quality product on the dominant platform is easier if your support the minor platforms as well.

Porting argument

Starting with Windows, Linux and macOS makes porting to XBox One, PS4 and Switch a LOT easier. There will be a clear separation between common and platform specific. Maybe you’ll never port to those platforms but if you do you’ll be ready.

Is it free?

Hell no. But if it were free it probably wouldn’t be worth doing. Machines take time to maintain, they cost money to run and purchase. Certainly macOS is an expensive platform to buy in to with it’s low end machines being expensive and very low end.

Over time there is a cost as well. With the transition from DX11 to DX12 and OpenGL to Vulkan I need to manage that. Unity and Unreal will ‘hopefully’ make this completely transparent to their users (but I doubt it). macOS moves pretty slowly but it routinely drops features where as Windows has only recently dropped some DOS features! This needs to be factored in to the long term maintenance costs of an engine and game.

Do you make your money back? It’s hard to say definitely which could mean no. Linux and macOS sales are low so the direct rewards are also low. However this needs to be weighed up against the increase in quality of the engine. For Smith and Winston right now I don’t think it’s been worth it financially (but it will probably make it’s money back on Launch), but mentally (peace of mind), morally and quality wise it’s been more than worth it.

Fanboy!

I am not a Linux fan boy. In fact I use FreeBSD, recommend FreeBSD and am a total FreeBSD fan boy. I’ll probably never ship a game on FreeBSD because that’s just not what it is for. Linux is however a desktop OS and a pretty good one so I see no reason not to support it.

Conclusion

People will tell you it’s not worth supporting the minor desktop OSes. My argument is that by doing it you will create a better quality product on the major OSes saving you money in testing and bug fixing for the majority of your users. I have nothing other than 25 years of commercial experience making games to back up this statement so feel free to ignore it.

Happy new year and good luck in what ever you decide to do.

Smith and Winston - Update #3

2019-01-02T00:00:00+00:00

Today we released the third update to Smith and Winston (only two more to go before it’s finished!). I’ll cover the new levels, bugfixes, tweaks and the future development in this post.

New Levels

There are three new levels in this release. First The Sky Ways. This level is a crazy mix of combat and puzzle. There is a clue in this picture if you ever make it this far :) Don’t worry, there isn’t any combat on these thin walkways… I don’ think ;)

Hanging Rock. Back in the forest zone of this weird ring world. Things are looking even more broken then before… spider crab things are everywhere! Watch your back.

Abandoned This is a weird place. A mixture of army base and shipping port. Hidden portals, dark passageways and eerie poorly placed street lights.

Tweaks

Weirdly we didn’t get everything right in the game up til now so we’ve made some tweaks after feedback from players:

Fixed fly forever ceiling bug - turns out that if you were in a cave with an invisible ceiling you could fly forever and basically scoot across all the lava and enemies that we deliberately put in the way to make the game hard. Very naughty of you so we’ve fixed that!
We tweaked the elite enemies so they aren’t SO hard to kill. They basically had really high health and soaked up way too many bullets.
Reworked the bunker boss as it was too hard and actually became a bit tedious. We don’t want that so we’ve removed a stage, slowed down the enemy spawning and reduced his health a little.
Added sound effects to doors and lifts in the early levels.

Bug Fixes

In Deep Down Again we reduced the fog a little as the enemies were too hard to see.
Fixed Proximity mines sometimes reappearing after death but invisible.
Fixed Proximity mines camera shake so it doesn’t feel like you’re going to spew.
Fix for the player flying/hovering forever. If you flew up and as you were about to reach the floor you dashes and the next frame there was a certain type of voxel you would fly/hover forever until death.
Fix a bug were rebuilding level collision/models occurred multiple times because dazza is an idiot. (helps low end machines a lot)
Removed some annoying debug prints that slowed down windows builds a bit.

Future Development

In general development is going really well. The engine is basically stable now across Windows, Linux and macOS. Crashes are pretty rare which is always a good thing. This leaves more time for dazza to concentrate on game play, bosses and polishing a few things here and there. The levels are on track for release which is great. We were both ill over the xmas period so we’re running a few weeks behind schedule for polish and sound which is very annoying. Nothing we can do about that though.

We’re really struggling getting all the sound design done for the levels unfortunately and we’re a good four weeks behind on that.

Steam is so nearly ready to go. The game is approved by Steam and we’re ready to go in to Early Access. We’re just getting together a decent press kit so that hopefully we can get some press coverage for the game. I’ll just reiterate, if you’ve bought the game on itch.io you are entitled to a Steam key if you want one (we get asked this a lot :) ). We’ll keep you updated on the status of Steam. Hopefully you’ll know because of all the press: billboards, sky writing, logo on the moon, maybe a mention from some streamer I’ve never heard of… or just an email from us :)

Anyway, thanks for reading this far, the two of us are really thankful for all your feedback, kind words and patience while we try and make Smith and Winston as good as possible.

Have a great new year and keep on exploring and destroying.

Smith and Winston - Update #2

2018-12-05T00:00:00+00:00

Earlier than promised (by only a day) we’ve released the second update to Smith and Winston. Of course we’re really excited to hear what you guys think of the new levels including another secret level! Can you find it?

Get lost in the ancient underground temples but watch out for the killer space slugs. They take a lot of hits to kill.

Get up high and survey the landscape, jumping to the left of me, enemies to the right and here I am stuck in the middle with… is that a weapon power up down there!

When you’ve spent the evening shooting, plundering and exploring why not take a trip to the Chalk and Flowers level. The sun is setting, the birds are chirping and the blossoms are blossoming. There’s loot to be plundered here but it’s not where you expect.

Of course this release comes with a hosts of bug fixes including a lovely crash bug that we’ve been hunting for a few years, an AI bug where they just jumped off the world for no reason and a sound bug where enemies would explode forever deafening everyone for one hundred miles. Thanks to everyone who reported bugs and gave feedback on here, Reddit and Steam.

If you don’t feel like buying the game then follow us on itch for more update, Wishlist Smith and Winston on Steam or just check back here for info. There will be more news about steam soon. If you buy on here, we’ll give you a Steam key when the game releases on Steam.

Explore and Destroy.

Smith and Winston - Update #1

2018-11-02T00:00:00+00:00

We are amazingly happy to release Update #1 to everyone! It’s been two weeks of hard work, over 50 bug fixes, three new levels and a metric tonne of tweaking and balancing. Thanks for all the feedback from the initial release. Smith and Winston is now EVEN better.

So what is in the update?

Bunker Boss

You meet this guy right at the end of this update. He’s a recurring enemy, he lives in a bunker that belches out fumes and he protects himself with enemy spawners and massive cannons. Do you think you’ve got what it takes to beat him?

Swamp of the Reanimated

So who would have though that near a damp, dinghy womble infested swamp there would a maze of lava caverns hiding a few secrets? Games designers, that’s who. Everyone loves a lava section and if you don’t… well you might not even find it.

Candy

It’s not all dark and dingy in the shattered ring world future of dooooom. Candy is so colourful it might make you sick… after Halloween I think we can all relate to that… even us brits who have inexplicably started to dress as ghouls for one day of the year!

Why oh why is Smithy sitting in that strange hole though…

Maze

Leave the dark and enter the light after the maze. This place must have been lovely before… well before it all went wrong.

Collectable Helmets

That’s it till the next update

We’re, of course, very proud of what we’re releasing to all you lucky people today. To be honest though it’s so much better for all the feedback you’ve given us. From tweaking the controls to huge holes in the level design you’ve been fantastic partners in the making of Smith and Winston.

The next update is scheduled for the 6th of December, more levels, more destruction and more fun.

Thanks… now go forth… EXPLORE AND DESTROY

Smith and Winston - First Access

2018-10-12T00:00:00+01:00

Well it’s been a long long four years and Smith and Winston isn’t done yet but it is a major milestone for us. We’re opening the game up for first-access.

Why not just wait and release it? Well when you’ve worked on something for so long there comes a time when you just need to hear what other people think. Charlie and I love playing the game but what do YOU think?

Why did we go with itch.io? We’re a two person team. One coder and one artist. To be frank if we launch on steam we’re terrified that we’ll be mobbed with bugs and complaints. Itch just seems a little bit more human and a little more kind… if that makes sense. Sure we’ll release on Steam later but for us itch.io seems like the right place to be.

Together, Chalie and I have over fourty years of game development experience. We’re not spring chickens by any means but we’re committed to releasing Smith and Winston and making it the best game we can.

Here are some images of Smith and Winston over the years:

We originally prototyped an Ikari Warrior game in Unity in 2015. It was terrible but we enjoyed the Ikari Warrior elements.

For some reason we decided that we’d prototype a voxel version of the game, this is an early prototype in Unity or maybe Unreal engine. It looks good but both Unreal and Unity were a bit too slow for the size of voxels we wanted (they were slow at collision mesh update basically). I’d written many engines over the year so I set about making my own “quick” engine.

Again from 2015 maybe 16, we’ve got basic combat, the camera can’t rotate but it already looks REALLY good (if I say so myself).

Another six months and we’ve polished the lighting and the cracking that grenades can cause in the voxel landscape.

Here, about a year ago (give or take) the voxel version, code and design really start to hit their stride. Now it’s amazing to play and it’s starting to get challenging.

All these pictures are on our imgur account so you can see how we got here: Imgur

Smith and Winston mentioned in Kotaku UK - We’re sexy!

2018-01-14T00:00:00+00:00

We’re not that great at promoting Smith and Winston. It’s not finished yet, so we’re very busy, but we also know that we need to get as many people to see the game before it’s done so they know it exists. So we are very grateful to Kotaku and Laura Kate Dale for including is on a big list of British Games coming out in 2018… wahooo. There are some great games on the list as well as ours so why not have a wonder through the future of british games.

Hopefully we can get the game done ASAP and get some feed back from real humans. Not just our exhausted dev team of me and Charlie ;)

New Year, new HUD

2018-01-10T00:00:00+00:00

One of the elements we’ve iterated on the most in Smith and Winston has been the HUD. It really is hard to convey all the information in a quick and timely manner to the player. This is especially true for a game with as fast action as Smith and Winston. Here is out latest revision.

Designing it so that it worked in both light and dark scenarios was also a challenge.

It really feel like it’s starting to come together now though so that’s good.

Installing MacOS 9.2 on my Power Mac G4 (Sawtooth)

2018-01-07T00:00:00+00:00

For reasons that are obscure even to me, I felt the desire to unarchive some old DAT tapes that have been sitting around my house for the last 20 years. They were backed up with Retrospect backup about 20 years ago on an old PowerMac 7100/80 when I worked at a company called The Multimedia Corporation (now defunct).

I looked at the DDS2 tapes on FreeBSD and it said they were blank (using tar) but I couldn’t be sure that Retrospect wasn’t writing to the tape in a non standard way. So I could either look at the tape as a raw character device and maybe get some data from it or install OS9 and Retrospect on my old G4 and restore the tapes. The latter should be easier! Plus the thought of actually reverse engineering the Restrospect tape format (which is as far as I know completely undocumented) was a bit daunting. Especially if there was an easy out.

Installing OS 9 on to my G4 was a total PITA because of duff ISOs and a lot of duff information around. The ISO I found that did work for me was from WinWorld. It’s MacOS 9.2.1 and definitely supports that Sawtooth G4. I did a complete Universal install (you need to select custom install in the Mac OS installer). This does mean it boots a little more slowly but at least it works and you can move the drive to another OS9+ compatible computer and boot.

Before stumbling upon the WinWorld ISOs I tried many Toast files and it took me a long time to learn that you can’t burn them in OSX and have them boot on old world Macs. I also learned that you can’t turn them in to ISOs, burn them and have them boot in old world Macs. Many CDs lost their lives in the pursuit of this knowledge. I also learned that the netboot image floating around the internet boots but is unstable on my G4! I got to the point where I started to suspect my old G4 had a hardware fault. Despite it running flawlessly in OSX 10.4 is qould rarely run for more than 5 minutes in OS9.

Once MacOS was successfully installed I jumped through many hoops of SCSI<->SCSI connectivity and adapter to connect my DDS3 drive to my G4. I downloaded an old copy of Retrospect and… yep, you guessed it… the tapes were blank. Either I had blanked them, or I never backed up to these tapes or the tapes are dead and Retrospect is reporting them as blank (this seems unlikely to me). So anyway, lesson learned, just watch TV over christmas and don’t have plans! I know I have at least another 5 tapes somewhere in the house so I’ve left the setup ready to go if I ever stumble upon the tapes.

Here is an action shot, sometime around 3am on Christmas Day with my FreeBSD machine open so I can run a SCSI cable to the internals of the Mac. Another partial success!

Interesting to me is that my G4 has an accelerator in it. I totally forgot that I bought this. It takes it from 450MHz to 1.4GHz… yes giga! It was one of the factors that I thought could be causing the various OS9 installs to crash. Along with swapping out the HD, Memory, all PCI cards, CD-ROM and the AGP card. I had a lot of trouble finding the spec sheet on Other World Computings website so it’s here but the original from OWC is here (for now).

Hope this helps poor soul who wants to use their G4.

As an aside I discovered on Boxing day that my G5 no longer turns on and just clicks when the on button is pressed. I have no idea where to start diagnosing that. A future blog post I guess

dazza

Mounting USB sticks autmatically in FreeBSD

2018-01-01T10:20:00+00:00

Getting USB devices to automount in FreeBSD is harder and more complex that it needs to be. All the parts are there but what makes it hard is that it’s changed over the years so you can get yourself in a right pickle. Here is what I did to get it working with FreeBSD 11.1-RELASE. If you follow the handbook you end up with USB sticks that mount but are readonly and can’t be unmounted by a non-root user.

Firstly follow FreeBSD Handbook section on USB sections 17.4.1 and 17.4.2 BUT DO NOT uncomment the

1	`/media -media -nosuid`

line in /etc/auto_master and DO NOT uncomment the lines in devd.conf (or move them above the comment as the file says).

Next install Freshports/automount from ports or pkg.

1	`pkg install automount`

Automount is going to be called by devfs when a USB device is detected on insertion. The installer installs the devd additions in to /usr/local/etc/devd/automount_devd.conf for you (which is why you don’t need to do it yourself).

Automount itself needs to be configured. So cp /usr/local/etc/automount.conf.sample /usr/local/etc/automount.conf and then open it in your favourite editor.

USERUMOUNT=YES
ATIME=NO
REMOVEDIRS=YES
FM="thunar"
USER="USERNAME"
ENCODING=pl_PL.ISO8859-2
CODEPAGE=cp852

I changed USER to my username and FM to launch thunar when a USB is inserted. In a multi user system this isn’t ideal as there is only one user. My only thought on this is to create a user called USBThumb that can’t login and add all users to the USBThumb group. I’ve not done this but you can see where I am going with it.

Anyway, this should get you going. Hope it helps someone.

Merry XMas and a happy new year

2017-12-25T00:00:00+00:00

We took one of the secret levels from Smith and Winston and played with it for a few days to make a Christmas style level. We’ll probably hide this level somewhere in the game, maybe it will only appear on Christmas day?

Anyway, we hope everyone has a really happy break.

Connecting to a m0n0wall box over serial

2017-08-20T00:00:00+01:00

I dug up my old m0n0wall box today and wanted to see what was on it. It’s an old WRAP embedded system from PC Engines. It doesn’t have any video out. I didn’t want to connect it to my network as it’s likely to have the ipaddress 192.168.1.1 and that would cause conflicts with my pfSense box.

So I connected up a serial cable, connected to the box using

sudo cu -l /dev/cuaU0 -s 38400

The docs say that the speed should be 115200 but I found that didn’t work (even after resetting the box back to factory defaults).

This is really just a blog post for me. In a few years I will stumble across this box again and need to know how to connect to it :)

Incase anyone stumbles across this on google looking for a FreeBSD compatible USB to serial. I used a USB to Serial cable that I had lying around. I worked fine with FreeBSD 11. The cable appeared as

Prolific Technology Inc. USB-Serial Controller D

in dmesg you see this:

ugen0.2: <Prolific Technology Inc. USB-Serial Controller D> at usbus0
uplcom0 on uhub4
uplcom0: <Prolific Technology Inc. USB-Serial Controller D, class 0/0, rev 1.10/4.00, addr 2> on usbus0

For reference here is the serial output of the WRAP box as it boots. This box used to be a lovely firewall but m0n0wall’s retirement and it’s slow processor forced an upgrade to a more powerful box.

PC Engines WRAP.1C/1D/1E v1.11
640 KB Base Memory
130048 KB Extended Memory

01F0 Master 848A Flash Card                              
Phys C/H/S 984/16/8 Log C/H/S 984/16/8
Consoles: internal video/keyboard  
BIOS drive C: is disk0
BIOS 640kB/130048kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(root@mb64.neon1.net, Thu Jul 23 20:56:49 CEST 2009)
/kernel text=0x368050 data=0x3d474+0x22ef4 syms=[0x4+0x47050+0x4+0x5acc6]
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.4-RELEASE-p5 #0: Sun Jan  9 22:24:57 CET 2011
    root@mb64.neon1.net:/usr/src/sys/i386/compile/M0N0WALL_EMBEDDED
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Geode(TM) Integrated Processor by National Semi (266.65-MHz 586-class CPU)
  Origin = "Geode by NSC"  Id = 0x540  Stepping = 0
  Features=0x808131<FPU,TSC,MSR,CX8,CMOV,MMX>
real memory  = 134217728 (128 MB)
avail memory = 105136128 (100 MB)
wlan: mac acl policy registered
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
cpu0 on motherboard
pcib0: <Host to PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
ath0: <Atheros 5212> mem 0x80000000-0x8000ffff irq 12 at device 13.0 on pci0
ath0: Ethernet address: 00:1b:b1:02:bf:0e
ath0: mac 5.9 phy 4.3 radio 3.6
sis0: <NatSemi DP8381[56] 10/100BaseTX> port 0x1000-0x10ff mem 0x80040000-0x80040fff irq 10 at device 14.0 on pci0
sis0: Silicon Revision: DP83816A
miibus0: <MII bus> on sis0
nsphyter0: <DP83815 10/100 media interface> on miibus0
nsphyter0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis0: Ethernet address: 00:0d:b9:03:f1:74
sis1: <NatSemi DP8381[56] 10/100BaseTX> port 0x1400-0x14ff mem 0x80080000-0x80080fff irq 9 at device 15.0 on pci0
sis1: Silicon Revision: DP83816A
miibus1: <MII bus> on sis1
nsphyter1: <DP83815 10/100 media interface> on miibus1
nsphyter1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis1: Ethernet address: 00:0d:b9:03:f1:75
sis2: <NatSemi DP8381[56] 10/100BaseTX> port 0x1800-0x18ff mem 0x800c0000-0x800c0fff irq 11 at device 16.0 on pci0
sis2: Silicon Revision: DP83816A
miibus2: <MII bus> on sis2
nsphyter2: <DP83815 10/100 media interface> on miibus2
nsphyter2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis2: Ethernet address: 00:0d:b9:03:f1:76
Geode PC Engines WRAP.1C/1D/1E v1.tinyBIOS V1.4a (C)1997-2005
isab0: <PCI-ISA bridge> port 0xf400-0xf43f,0xf600-0xf63f at device 18.0 on pci0
isa0: <ISA bus> on isab0
pci0: <bridge> at device 18.1 (no driver attached)
atapci0: <National Geode SC1100 ATA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 18.2 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <multimedia, audio> at device 18.3 (no driver attached)
Timecounter "Geode" frequency 27000000 Hz quality 1000
pci0: <bridge> at device 18.5 (no driver attached)
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xe0000-0xe7fff on isa0
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
RTC BIOS diagnostic error 80<clock_battery>
Timecounters tick every 1.000 msec
Fast IPsec: Initialized Security Association Processing.
IP Filter: v4.1.33 initialized.  Default = block all, Logging = enabled
md0: Preloaded image </mfsroot> 16777216 bytes at 0xc086b0e8
ad0: FAILURE - SETFEATURES SET TRANSFER MODE status=51<READY,DSC,ERROR> error=4<ABORTED>
ad0: 61MB <Flash Card 2N2-1205> at ata0-master BIOSPIO
Trying to mount root from ufs:/dev/md0
kern.coredump: 1 -> 0
Found configuration on ad0.
sis0: link state changed to DOWN
sis1: link state changed to DOWN
sis2: link state changed to DOWN
Initializing timezone... done
Configuring firewall... done
Configuring LAN interface... done
Configuring WAN interface... done
Starting syslog service... done
Starting webGUI... done
Starting DNS forwarder... done
Starting DHCP service... done
Starting NTP client... done


*** This is m0n0wall, version 1.33

Hope this helps some lost cave deweller somewhere.

dazza

FreeBSD 11 on the Acer Aspire One ZG5

2017-08-19T10:20:00+01:00

For no reason other than boredom I bought an Acer Aspire One ZG5 notebook for £30 on ebay. I wanted to install FreeBSD on it and with and little google-fu I checked that it was indeed compatible with FreeBSD 11. Auction ended, off I go installing the best OS in the world!

To The Desktop!

Installation from a USB stick went without hitch. I then largely followed the excellent guide here to get the machine to a reasonable desktop state in about 30 minutes using pkg. I went with the more lightweight XFCE desktop and the little Atom CPU seems to run it just fine. I initially intended to run Window Maker for that good old NeXT feeling but after a while I had to admit that there were a lot of packages that were not supported by the FreeBSD ports team and I needed something a little more complete. Window Maker ran really really fast though so I wish I had the time to upgrade the packages or even become the pkg maintainer for the FreeBSD project.

System Tweaks/Settings

I added the following tweaks to /etc/sysctl.conf

hw.usb.no_shutdown_wait=1
hw.sdhci.enable_msi=0
hw.acpi.lid_switch_state=s3
hint.apic.0.disabled=1

Unfortunately hw.sdhci.enable_msi doesn’t make the SD Card reader work but it does stop a storm of messages about the controller timing out if someone does insert an SD Card.

The default lid state hw.acpi.lid_switch_state was None so I changed the to s3. On closing the lid the machine went to sleep but upon opening the lid the machine crashed or best case the screen stayed black. Setting hint.apic.0.disabled and adding acpi_video_load="YES" to /etc/rc.conf seemed to fix it but I am not sure why. There is a lot of black magic around the buggy drivers/BIOS/microcode for ACPI compliant hardware.

I added the following to /etc/rc.conf to load the Intel driver and to get the acpi subsystem to work with the screen.

i915kms_load="YES"

The webcam didn’t work out of the box, to get the system to recognise it I install webcamd and added this

webcamd_enable="YES"
webcamd_0_flags="-N Sonix-Technology-Co---Ltd--USB-2-0-Camera -S unknown -M 0"

to /etc/rc.conf restarted devd and used pwcview to preview the glorious 320x240 video it generated at about 15fps. I commented out the settings in rc.conf in the end. Who wants a webcam that bad!

Xrandr - External Monitors

Setting up an external display worked with two of my spare monitors. An NEC 17” and an LG (Goldstar) 22”. Pretty seamless as well.

The two desktops, one placed below the other.

Stripped down apps

The Aspire One isn’t powerful enough to run a full browser like Firefox or Chromium so I installed Midori for web browsing. Midori uses about one third of the memory and renders nearly all pages I go to. For email Thunderbird was just as heavy weight as Firefox so I installed Claws Mail for IMAP mail access. I actually use Mutt with some custom commands but a lot of email comes through as HTML only and it can get annoying reading it in Mutt.

Conclusion

Overall the installation went very smoothly, no crashes but of course a lot of tweaking needed to be done especially around sleep states and the peripherals.

FYI I am running this version of FreeBSD11

>uname -a
FreeBSD mitch 11.1-RELEASE-p1 FreeBSD 11.1-RELEASE-p1 #0: Wed Aug  9 11:17:49 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  i386

Overall I’m really happy with the little netbook. It’s very light, I spent £30 on the pristine notebook, £8 on another GB of memory to max it at a whopping 1.5GB of RAM and £10 on a battery that gives it about four hours of reasonable use.

If an SSD becomes free from one of my other machines I’ll pop that in but it’s not worth buying an SSD for this machine at the moment. It’s fine with it’s 120GB 5400RPM drive.

The only downside I can see at the moment is that the fan is a little noisy. I’m not sure how to fix that. I could replace the fan but all small fans have a high pitch whine. I’ve eneabled powerd and I can see that the CPU is being slowed down so I guess I’ll have to live with it

Hope this helps someone else who wants a small, cheap, light weight almost throw away laptop.

Update 171226 - Turns out installing AcerHDF-kmod fixes the noisy fan which makes a HUGE difference.

dazza.

In Defence of Writing Your Own Engine

2017-07-14T00:00:00+01:00

Here’s to the crazy ones, the misfits, the trouble makers, the engine-eers! - dazza

After reading this excellent blog from ZeroEqualsFalse I thought I’d counter with a defence for writing and deploying your own engine.

I’ve been writing “engines” for 20 years both professionally and for fun. I am in no way a Carmack and I am under no illusion that I can compete with all of Unreal or Unity on all fronts but…

Pushing it

If your game is in any way pushing the limits you can’t stick with Unity and often even Unreal is not ideal. For Smith and Winston we need to regenerate detailed visible and collision meshes VERY quickly as the player can destroy nearly anything in the game.

We prototyped the game in Unity but it had slow mesh creation times and quite frankly shockingly bad collision mesh insertion times. By writing our own engine we managed to get orders of magnitude speed increases over Unity and about 10% faster than Unreal. Write a plugin you say! There you go assuming most of the speed boost didn’t come from not working through multiple layers of abstraction that general engines necessarily have.

This may have changed now, I see there are some excellent voxel plugins both Unity and Unreal but my point still stands in general.

It’s part of the fun

Part of what attracted me to making games was the tech side. There are loads of people who have been enabled to make games because the bar has been lowered and this is a great thing but there are many people like me that love the tech, love the challenge, love the low level, love the bus speed usage, love the lack of memory fragmentation, love the aligned cache access, love the parallel thread usage as well as love making games. If you’re one of them go for it!

We need skills

ZeroEqualsFalse make an excellent point that writing an engine is a great learning experience. You’ll only really understand engines by writing one. They all have their own quirks, good points and bad points. But why would you waste your time writing an engine when Unity and Unreal are quite frankly great engines? As an industry we need these skillz to be honed and we need people that understand how to finish an engine as well as a game.

Taking an engine to market, getting it through QA, TRC, localisation and deploying it on a myriad of platforms is a valuable skill and a skill that is rapidly disappearing. During the “democratisation of game making” that has been going on for the last 10 years people have been actively discouraged from learning these valuable skills. Someone has to write the bloody engines of the future!

Plurality

It’s quite frankly not healthy for our industry that there are two dominant engines (Unity and Unreal). There are other great engines (Godot being a great example) that are largely pushed to the margins. The industry already suffers from Autodesk dominating the creation pipeline with Maya and 3DSMax. At some point one of those engines will collapse under it’s own weight, competition or market forces. This will leave a huge hole in the scene.

Its actually ‘not that hard’…

but can be time consuming. 20 years ago when I wrote my first engine (as opposed to just a game) every single line of code was mine. It had custom string, vector, matrix, model, collision, physics, sound, rasteriser, input… you get the point. I wrote it all in C and “C with classes”!

Nowadays you really don’t want to be doing that and you don’t need to! Here are the libraries that I use in the Smith and Winston codebase

PolyVox - excellent voxel library, I didn’t even write the voxel code!
Bullet - world class physics (soft and rigid) and collision library
BGFX - world class low overhead rendering API with DX9/11/12, OpenGL, Vulkan and Metal backends.
C++11/STL - night and day compared to old C++. Basically cross platform code is way easier to write now with MSVC/GCC and Clang.
CMake - meta makefiles system that’s arcane but makes cross platform project management so much easier.
OpenSteer - navigation library (bit dated but very usable)
GLM - video game maths and maths utility library. Apes the GLSL syntax.
LibZlib/lz4 - open, free and very squashy.
SDL - input, windows, sound, back massages
FMOD - For all your sound needs OK, it’s not ‘free’. But it is basically free for all indie games and it’s amazingly good. It took two days to integrate in to my codebase!
Lua/LuaJIT - you are not going to get a language as fast and expressive as Lua that fits in < 1MB of memory. Scripting for grownups!
stb_* great collection of utility functions/classes by Sean Barret
plus countless other small bits and bobs

Essentially my engine is really just the glue between all of these off the shelf, highly polished and stable libraries.

For an editor just use Qt/WxWidgets or Mono. Integrate your engine with it and you’re most of the way to having a very usable editor.

You’re not really writing my own engine because you’re using all these great components.

Well if that’s true you’re not making sprites unless you’re typing in HEX.

To conclude

So it’s not for everyone and it’s great that more people can create all the varied games that we see and love but don’t discourage the future Carmacks of the world. GO FOR IT!

Whatever and how ever you decide to progress, be excellent to each other and PARTY ON, DUDES!.

have fun, dazza

updates

170723 - added facebook link
170723 - added links to the libs I mentioned as that’s how the internet is supposed to work.
170720 - This post generated an interesting discussion on Reddit.
220930 - removed links to smith and winston that are dead.

How to stick at it: Three years and still going strong!

2017-07-11T00:00:00+01:00

To celebrate (and commiserate) the fact that I’ve been working on Smith and Winston for three years I thought I’d answer the most commonly asked question: How do you stick at it for so long? Some of this applies to anything in life, but it’s mostly about making games as that’s all I do.

TL;DR: No one has a magic bullet, no one is especially driven, no one has an easy ride, it’s all about hard work and sacrifice.

The movies are only there to show how awesome Smith and Winston is, you can ignore them ;)

Stubbornness

Firstly you’ve got to be little bit stubborn. Life is full of distractions and you need to be able to park new ideas, new experiences and sometimes friends to focus on your goal. Telling people you love that you need to sit on your own for hours on end doing something they don’t understand is hard. For me it’s easy because quite frankly I’m stubborn and maybe selfish but…

Sacrifice

Something has to give and anything worth doing comes at a cost. Whether it’s money, time, relationships or health it’s all going to come under strain. Balancing these things is probably the biggest challenge. Never sacrifice health, money is easy to sacrifice to a point and sometimes you have to accept relationships are either not supportive of your goal or too expensive to maintain. Whatever you do, understand what you’re sacrificing and make sure you understand the cost/gain ratio. Sometimes it’s better to sacrifice the project totally and move on.

Routine

It doesn’t matter how driven you are: every day you don’t do something you are one step closer to never doing it again. It’s inspiration and excitement that starts a project but it’s discipline and hard work that finishes it. The initial excitement dissipates all too quickly and you are left with the long slog. No one is exempt from this.

Time

Time is indeed a cruel mistress. You need to work effectively to make progress. Install Rescue Time and learn how you work. Balance this with the fact that you are not a machine. Rescue Time will tell you when you are not working effectively. Learn why, if you’re bored shift to something more interesting that needs doing but don’t forget to complete the boring thing. If you’re tired then take a nap and come back. If you’re totally lacking inspiration take time off, play some games, get inspired. Come back. Sometimes you just need for force yourself to work.

Nurture the initial spark

Smith and Winston was inspired by memories of playing Ikari Warriors on my Commodore 64. We’d just canned our previous game, I took a few weeks off and was seriously considering going to work for megacorp to make mucho-dinero. Replaying Ikari Warriors totally rejuvenated my energy for making games. I go back to it frequently to remind me why I started this marathon. Don’t expect every day to be fun. It won’t be. Famously the last 10% of any project takes 90% of the time and it’s mostly not fun. Remember the initial spark and be stubborn. Never forget to stop playing.

Understanding criticism

As soon as people close to you know you’re doing something off the beaten track they will offer support and criticism. Both are vital but both are double edged.

You need to see all unsolicitied comments through the prism of the person making it. Not everyone has your best interests at heart and some people (hello mum) can’t see anything bad in you. Everyone’s opinion is valid and should be considered but some are “more valid”.

Learning to take it professionally and NOT personally is something even the longest, greyest beards of the games industry are still learning to do.

Orders of magnitude

A player might get 100 hours of gameplay from your game (if it’s huge or repetative) but you will see your game for thousands of hours and it will look boring stale and repetitive. Resist the urge to change, revise and upgrade constantly. Always seek to cut and hone.

Seeking constant resssurance

If you rely on constant reassurance from your peers you’ll hit the rocks. People spend a lot of their time and energy blogging every little detail and live streaming. This offers instant reward for minimal effort. You need to reward true effort to sustain yourself. There is a lot to be said for building up the resillience of the lonely artist working through problems without the safety net of your peers holding your hands. They won’t always be there but you always will be.

This is my game Smith and Winston, it’s awesome

updates:

170723 - added link to Smith and Winston below gif.
170723 - changed title from “Three Year and Still Going Strong” to “How to Stick at it: Three Year and Still Going Strong” to make it more clear.

Smith and Winston Alpha 4

2016-08-03T00:00:00+01:00

9 months!

It’s been nine months since the last video update. Of course we’ve been busy working on Smith and Winston. Here is the latest video:

We’ve added:

Depth of field effect to soften the distant voxels.
Complete change of enemy and weapon strategy to bring the combat closer to the player.
Weapons: You now get a pea shooter, sub machine gun, rocket launcher and assault rifle as well as your trusty grenades.
Enemies throw grenades.
Island level design.
Skybox with cubey voxel goodness.
Clouds!
Many more particle effects on explosions, birds, butterflys and flowers.
A more functional if still not very pretty HUD.
A front end (you don’t know how nice it is to have a front end!)
removed nearly 1GB of memory leaks!
2 player co-op working
Explosions cause explosions.

Island Level Design

With the previous monolithic level design we were finding it really time consuming balancing the levels. If we wanted to change the structure of the level it took a long time to copy and paste the voxels about. Now we can move the islands around and add bridges trivially. This is a big win for a two person team.

Another problem we’ve had is on low end machines the monolithic levels were too expensive to modify during explosions. One of the downsides of voxels is that they can take a lot of memory to store and slow machines crawled modifying the voxels, rebuilding the models and collision meshes. Monolithic levels were also taking up 1.5GB of memory. Our new levels take up about 0.8GB helping out the lower end machines.

Networking Folly.

I spent about six months adding networking to our codebase and in the end it was a failure. I didn’t have enough experience to pull it off and despite it ‘working’ on a LAN it was too dodgy for public consumption. This doesn’t mean it’s abandoned but it wont be added until after the game is released or maybe Smith and Winston 2. Replicating all the damage the player makes to the voxel world and keeping everyone in sync proved to be too much for my small brain to cope with :( The upside is that I’m making progress on the actual game play now.

Two player is great fun

Chained Explosions

I tweaked the explosion code so that it affects objects in the world, this allows you to chain up events so a grenade explodes a mine that blows up a barrel that blows up the enemy :) See how the proximity mine explosions cascade but also the turret is destroyed by the crab emeny exploding next to it.

Front End

Getting a front end, however simple, always makes a game feel more complete.

Steam Refund Policy

2015-06-04T00:00:00+01:00

If you haven’t heard, steam is offering a refund scheme to players.

Anyone can get a refund if

they’ve played the game for less than two hours.
they’ve owned the game for less than 14 days.
they ask for it.
they aren’t taking the piss by repeatedly asking for refunds.

At the end of the day players need a way to get their money back. Some games are a total abuse of steam and players. This is inevitable within any system and it’s only right to weed those ~~devs~~ practices out. While it may be possible for players to abuse the new system it’s never fair that it’s possible for developers to abuse players. So on the one hand I applaud the Steam team for confronting this poisoning of the well by the few bad devs and keeping the well clean for the rest of us. However I do have some concerns and it’s mostly about how developers will respond, not players:

14 days/Two hours

As an old man I regularly go 14 days where I can’t find two hours to play a game. As a youngster, especially, during the holidays, I had a seemingly infinite amount of time to play games so this would have been amazing. I would predict a lot of churn of the store as gamers churn through endless games trying them for two hours… however:

Refunds are designed to remove the risk from purchasing titles on Steam—not as a way to get free games. If it appears to us that you are abusing refunds, we may stop offering them to you. We do not consider it abuse to request a refund on a title that was purchased just before a sale and then immediately rebuying that title for the sale price.

So hopefully this will stop the churn, the heuristic that Steam uses can make or break the system.

DLC Crap

These new rules also apply to DLC. Some DLC is utter tripe and clearly there to take money from the weak or gullable. Getting refunds on this is totally appropriate. I’d actually rather see a much stricter system where if a DLC is constantly getting refunded it’s taken down from the store until the developer can justify it’s inclusion. Actually I’d rather see micro DLC’s banned but I’m old so I’ll ignore it.

Unintended Consequences

As a game dev I’ll admit that we always front load the gameplay and testing to the early stages of the game. We know that most gamers will not make it to the end boss and we want as many people as possible to have a great experience. With limited resources we priorities the areas of the game that most people will experience.

This doesn’t mean we don’t care about the end of the game but, being honest, it does get slightly less attention than the beginning.

However, for some, this hard limit of two hours will force them (or maybe even allow them) to only concentrate on the first two hours leaving the rest of the game a bit of a waste land.

How to stop developers doing this is a much harder problem. The only thing I can think is for players to review games and that’s pretty much how it works now.

What ever rules Steam enforce will be abused by some game developers and I believe that abuse is more detremental than any abuse that players can throw at Steam (not that I would condone any abuse by players against the ramen eating developers of the world).

Will this stop any dev using steam?

NO. Actually it will probably make steam a better place. Steam has all the players and a great community as well.

See how our game, Smith and Winston, is doing IndieDB.

Bullet Physics: Query objects within a volume

2015-03-27T00:00:00+00:00

I need to preform a query of the game world to find all Game Objects within a volume. Preferably a sphere but any convex shape would be good. I struggled with this for a few hours before realising it’s actually pretty easy. Bullet Physics is amazing but sometimes simple things aren’t that obvious.

My first approach was to use a sphere and btCollsiionWorld::contactTest. This worked^tm in that when I tested it a few months ago all was well but actually it only looks for collision shapes that overlap the boundary of the test shape. So a collision shape that was completely inside a test sphere would be missed. I guess the clue was i the name really.

Next up I tried a btCollisionWorld::convexSweepTest where the from and too transforms of the sweep were very close together. This seemed to return basically the same as btCollisionWorld::contactTest and it felt a little dirty as I didn’t really want a sweep test I just wanted to know what was in the volume of the sphere.

Now, I know, that bullet holds this information internally because it uses a tree structure of AABB to optimize it’s collision/physics system so it must have an accelerated way of modifiying this tree and keeping track of things that overlap or collide.

After stepping back for an hour or so I remembered that one of the collision shapes in Bullet is btPairCachingGhostObject and it is specifically designed to cache all the things that overlap it. I use it in the character controller for Smith and Winston. So here is the code is wrote to query a sphere of space for all objects that overlap this volume:

std::vector<GameObjectShPtr> results;

//[1]
btSphereShape sphere(radius);
btPairCachingGhostObject ghost;
btTransform xform;
xform.setOrigin(pos);
ghost.setCollisionShape(&sphere);
ghost.setWorldTransform(xform);
ObjectColMask *ocm = new ObjectColMask(colflags, ColFlags::kAll);
ghost.setUserPointer(ocm);

//[2]
mDynamicsWorld->addCollisionObject(&ghost);

//[3]
for (int i = 0; i < ghost.getNumOverlappingObjects(); i++){
    btCollisionObject *btco = ghost.getOverlappingObject(i);

    ObjectColMask *ocm = static_cast<ObjectColMask *>(btco->getUserPointer());

    if(ocm){
        ObjectCol *oc = ocm->UpcastToObjectCol();
        if(oc){
            GameObjectShPtr go = GOFromTransform(oc->GetTransform());
            results.push_back(go);
        }
    }
}

//[4]
mDynamicsWorld->removeCollisionObject(&ghost);
delete ocm;

return results;

Step 1

Create a sphere with the disired radius of the volume to test (I believe that any convex collision primitive will do, tri meshes wont work as they have no concept ot inside or outside).

Create a btPairCachingGhostObject set its collison shape to the sphere and set it’s collison flags. I’ve extended the collision flags with my own mechanism called an ObjectColMask.

Step 2

Add the ghost object to the collision world. The act of adding the objects causes it’s overlapping object list to be updated with, you’ve guessed it, all of it’s overlapping objects.

Step 3

Iterate over the overlapping objects doing your own magic to extract what ever information you need.

Step 4

Remove the Ghost Object from the collision world.

That’s it, pretty simple but not too obvious. Hope this helps someone else.

LuaJIT bit.band stopped working

2015-03-24T00:00:00+00:00

Came across an interesting feature/artifact of using LuaJIT today.

In Smith and Winston we use the bit package that LuaJIT provides to flag and mask collisions. The types of collision (for example world, bullet, vehicle) were hard coded on both the engine and Lua side. Today I moved toward the types being loaded from an external file by the engine/editor and injecting those in to the Lua global space for use by the scripts. No more hard coding, everything in one place, what’s not to like?

Well it didn’t work. Take the following code:

function Projectile:collision_trigger(othergo, localpos, 
                                      worldpos, otherlocalpos, 
                                      othercoltype, normal)

    --othercoltype = bit.tobit(othercoltype)
    --vox.print(string.format("bang %d\n", 23))
    if bit.band(othercoltype, vox.col.WORLD) ~= 0 then

        self:explosion(worldpos, normal)

    end
end

othertype should be a bit field showing the type of thing this GameObject has hit (for example WORLD). If it’s the world I want to cause an explosion at the impact location (worldpos) and move on. This didn’t work!

The weird things is that if I printed out othercoltype then the if statement worked!

The constant vox.col.WORLD is injected in to the Lua state using LuaBridge before anything gets a chance to run:

static int kWorld = ColFlags::kWorld;
getGlobalNamespace (L)
    .beginNamespace ("vox")
        .beginNamespace ("col")
            .addVariable("WORLD", &kWorld, false)

and has the value 2.

So it seems that bit.band(othercoltype, 2) doesn’t work so there is a difference between bitfields and number in LuaJIT. There is a clue to this in the LuaJIT bit documentation. The fact that it has a bit.tobit function implies there is something magic going on. To test this I changed the code to bit.band(othercoltype, bit.tobit(2)) and it started working. I have NO idea why printing the value worked but I that’s not a long term solution obviously.

So now I need to inject the normalized bit values in to the Lua state, not just the power of two numbers I thought.

Unfortunately this means playing with the Lua stack. I’ve been using LuaBridge to do all the heavy lifting up til now.

ColFlags &colflags = ColFlags::Get();
lua_createtable(L, 0, colflags.GetNumFlags());                  /* table */

for(size_t i = 0; i < colflags.GetNumFlags(); ++i){
    ColGroup colgroup = static_cast<ColGroup>(1 << i);
    std::string colname = colflags.NameFromFlag(colgroup);

    lua_pushstring(L, colname.c_str());                         /* table | name */

    lua_getglobal(L, "bit");                                    /* table | name | bit table */
    lua_getfield(L, -1, "tobit");                               /* table | name | bit table | tobit func */
    lua_pushinteger(L, colgroup);                               /* table | name | bit table | tobit func | colgroup*/
    lua_pcall(L, 1, 1, 0);                                      /* table | name | bit table | hexified colgroup */
    lua_remove (L, -2);                                         /* table | name | hexified colgroup */
    lua_settable(L, -3);                                        /* table */
}

lua_setglobal(L, "const");                                      /* <empty stack> */

In true Lua example fashion I’ve added comments showing the state of the Lua stack (relative to where the createtable code starts). I use a singleton called ColFlags that is initialized with the names and values of all the collision flags when the editor or engine is started up.

Now everything works, I’ve got all the collision flags defined in one place for both the Editor, Engine and Lua Scripts.

First Year of Smith and Winston

2015-03-10T00:00:00+00:00

I posted most of this to TIG Forums but thought I’d reproduce it here for people that don’t want to read a forum.

Smith and Winston is a twin stick shooter for Win, Mac, Linux and PS4.

S&W is Inspired by Ikari Warriors and Heavy Barrel, two great arcade games that are great fun to play even now.

Charlie (artist) and I (dazza the coder) have been working on Smith and Winston for just over a year now. We’ve come a long way in that time and this post is a sort of walk through time to show you where we are in the journey.

Initial Concept November 2013

We started out using Unity. Neither of us had used Unity before so it was a bit of a ride. After a few weeks we had a guy running around and shooting things. We weren’t really happy with the art style though, it just wasn’t coming together for us. Although Unity is great I didn’t really enjoy using it. Lots of crashes and lots of weirdness that as a coder I just wanted to fix.

Change of Art Direction Jan 2014

Initial 8bit character designs. We started to like what we were doing but how were we going to translate this in to a 3D game. We didn’t want to make it purely 2D like the originals but polygon work is very labour intensive and as a two man team that made us nervous.

Charlie creates these concept baddies using Qubicle (great voxel editor) and we seriously start to think we should make it a voxel based game (March 2014). I think in hindsight I was more reluctant than Charlie to make a voxel game. It seemed quite risky and Unity was already a bit if a risk.

Ditch Unity

I decide we should ditch Unity and run with our own code base so we can voxel this game up. While this seems bat shit it meant that I could say to Charlie what we definitely could and couldn’t do without worrying about Unity getting in the way. We both know it’s a lot of work though but it’s so exciting at the beginning isn’t it?

The editor starts and as always it’s a bit shit at the beginning. We really weren’t sure if we were doing the right thing dropping Unity but it seemed like a big risk to me and I knew I could get and engine/editor up and running and realise this game.

After a long few months (that’s coder speak for six months) the editor is starting to come together pretty well and we’re rocking some designs and playing some levels. The gap in writing the Editor and pulling the engine together gives Charlie a long time to pull concepts together and really nail the art style.

Here is a video showing: * how awful it looked in March 2014: https://www.youtube.com/watch?v=Pvriq8puKuU * looking better a few months later: https://www.youtube.com/watch?v=OjGY7pV3PUc * July 2014 and the editor is coming together: https://www.youtube.com/watch?v=hWTFmYu3k6g * August and the game is looking quite good: https://www.youtube.com/watch?v=NtrimC_TWe4

August - We decided on the Logo and name, set in stone now

Vertical Slice - New year Bosses

We started working on a ‘vertical slice’ in the new year. We wanted something to show potential publishers or maybe, if we can pluck up the courage, a kick starter campaign. For a vertical slice we needed a boss so we started to prototype one:

We want to show the varing terrains that the game will have so far we’ve got desert and jungle plus inside a military base type thing (ok not original but who cares!)

Grabbing keyboard input in Qt

2014-03-21T00:00:00+00:00

I’m currently writing an editor for our next game. We’ve used many tool in the past (Blender, Tiled & Mappy) to make games but we thought it was really time to invest in our own code base. This was heavily influenced by the great experience we had using Unity3D to prototype our next game.

I think you can see how much we were influenced by Unity3D in this shot.

Anyway, to write the editor I turned to Qt as a reliable cross platform library. I wanted to be able to play the game IN the editor so we could get some of the quick turn around for tweaking that you get in Unity3D.

The editor already uses SFML to read controllers and exposes this input to the Lua scripting system. However the SFML keyboard code (sf::keyboard) does not work because it’s not hooked in to the applications event loop as Qt owns (and basically hides away) the event loop.

To work around this I learned that you can install an Event Filter in the main application QWidget and filter out/redirect keyboard events. Using this I can stop keyboard short cuts from changing the editor state and pass the key presses to the Lua scripting system.

The Code.

So when the user presses the play button on the toolbar the code installs the Event Filter

QApplication::instance()->installEventFilter(mEventFilter);

and when the game stops, the user presses pause or stop the Event Filter is removed

QApplication::instance()->removeEventFilter(mEventFilter);

The event filter itself is pretty simple:

class EventFilter : public QObject{
    Q_OBJECT
public:

    virtual bool eventFilter(QObject *object, QEvent *event)override;
};

with the main actual Filter function looking like this

bool EventFilter::eventFilter(QObject */*object*/, QEvent *event){

    if (event->type() == QEvent::KeyPress) {
        QKeyEvent *keyEvent = static_cast<QKeyEvent *>(event);
        voxity::RegisterKeydown(QtKeyToSFML(keyEvent->key()));

        return true;
    }

    if (event->type() == QEvent::KeyRelease) {
        QKeyEvent *keyEvent = static_cast<QKeyEvent *>(event);
        voxity::RegisterKeyup(QtKeyToSFML(keyEvent->key()));
        return true;
    }

    if(event->type() == QEvent::Shortcut){
        QShortcutEvent *sc = static_cast<QShortcutEvent *>(event);
        const QKeySequence &ks = sc->key();

        voxity::RegisterKeydown(QtKeyToSFML(sc->key()[0]));
        return true;
    }

    return false;
}

If the filter function returns true then the event should be ignored by the rest of the application and the event is discarded. Otherwise it carries on it’s way.

The Qt docs talk about filter functions slowing down the app but it doesn’t seem to make much difference to our editor. It’s only installed when the game us running anyway so most of the load is in running the game.

Basically if it’s a keypress/keyrelease event the RegisterKeydown or RegisterKeyup is called. This records the keypress in a way that the Lua scripting system can pick up that a key is down or up. The Qt keycode is converted to an SFML code as the game runtime (when it running outside of the editor) will just be using SFML.

The one quirk is if the event is a Shortcut. The editor has many single keys that map to actions (for example w activates translate mode and e activates rotate mode). These come through as shortcuts NOT keypresses. This hints that there is a lot of code beneath this filter playing with the raw input. For the initial keypress of a shortcut a Shortcut event is triggered but the release of the key comes through as a KeyRelease event.

Not Perfect Though

This scheme doesn’t seem to intercept CMD-Q, CMD-P etc events that are assumed by all apps to be Quit and Print. Further investigation may reveal how they can be intercepted but I don’t need to grab them at the moment.

I hope this helps someone.

Why I Chose Not To Use Unity3D

2014-03-21T00:00:00+00:00

I evaluated Unity3D for a few months before deciding NOT to use it. It’s a really, really great tool and it was a tough decision. Ultimately I felt like there was more resitance to steady progress using Unity3D over extending/upgrading my own code. Here is a rough guide to why I decided not to use it.

C# is not a natural AI language

Now for many this isn’t an issue. C# is a great language. Compared to Lua though C# is too verbose and too cumbersome (not as cumbersome as C++ but close). It just takes too long to express simple things in C#. The screen is filled with too many curly braces. A lot of the verboseness comes from C# strong typing. Strong typing is a great feature and many bugs are caught early using it. However for me the slowest bit of writing a game is writing the scripting/behaviours/AI. It’s also the bit I’m least comfortable with as it’s all about nuance and anything that resists quick change and ‘play’ is a hinderance.

Of course you can use Boo and Javascript but it’s really very clear that they are second class citizens in the Unity3D world compared to C#.

Crash after crash (at least at the start)

For the first two weeks of using Unity3D I just got too many crashes in the editor. Most were to do with importing animations. I reported a bug and nothing really happened about it. The bugs were OS X specific (the same steps didn’t crash on Windows XP or 7). Once I’d got through that animation rough patch I only had the odd crash, nothing I’m not used to with my own software ;)

Most of the crashes I could work around by using windows or chosing a different path through the editor. This left a bad vibe with me though. What other bugs might be lurking within. Probability says “not many” since so many other devs are using Unity3D, but what lurks beneath?

Community

Unity3D’s strength comes from it’s flexibility and price point. It just can’t be beaten at the moment… apart from the forums. I imagine since going ‘free’ Unity3D’s forums have taken a real nose dive. There is frequent incivility, frequent arguments about the ‘best way to do something’ and no one with authority seems to step in.

I realise this is the internet and every asshat with a keyboard is Carmack but I don’t want to wade through Asshat Carmack stupidity when I am stuck.

Lack of Control

Personally I like to control the code that I work with. I’ve been writing games for thirty years and I’ve always controlled the code. While I enjoyed making a prototype game and I learned a hell of a lot I was nervous being this far from the metal.

Extending Unity3D

The game I was prototyping needed voxels. So I looked at how to extend Unity3D with my own type of mesh and collision primitives. C# is a poor scripting language but it’s a great systems language so I thought this would be easy, maybe there was a good module I could buy?

I really struggled to get an overview of the system and how to extend it. There are bits and pieces all over the place but I wanted one canonical document on how to extend the codebase. I wanted to interface Polyvox with Unity3D (something they would later announce they were doing anyway). Once again there was too much resistance stopping me from making progress.

I looked at buying modules but at the time (late 2013) there didn’t seem to be any modules I could buy to get me over this hump. No modules offered an evaluation period. I had very specific needs of speed and voxel density, I didn’t want to pay $2000 for something that didn’t meet my requirements. Since I wasn’t cloning Minecraft and didn’t need an infinite draw distance I was worried I was too niche.

The HUD Code

Oh my god, the immediate mode HUD stuff is bloody rubbish. I know this is a well documented problem and it’s being fixed but how long is it taking and how many devs have complained and oh my god how much of a PITA is it. Compared to the rest of the code base this is undergraduate quality.

Controllers

The way unity deals with controllers is amazingly bad. Of all the interfaces they provide this is the one where there is no justitifcation for such a bad interface. It makes dealing with PS3/4 and XBox controllers a real PITA.

In Conclusion

Everyone writing games should spend some time using Unity3D. It’s an amazing piece of software written by an amazing group of people. If you’re just starting out making games, have limited time or limited budget then it’s basically a no-brainer. If however you have a niche technical angle, a shit load of experience, a codebase you can work with then, well it’s not quite so clear.

Many of the stress points in making a game come from the polish end 10% (that famously takes 90% of the time). Unity does a lot to empower the non-coders in a team to polish. For me as a coder it doesn’t add much and may even hold me back by pushing C# and limiting extension.

Ultimately for me, as an engine coder, the crash bugs just gave me the willies. I’ve spent many a long night fixing crash bugs and burning CD’s for, or uploading to, Sony, Nintendo and Microsoft and not being in control just adds too much stress. Despite knowing many devs are happy with that I’m personally not happy with that lack of control.

Integrating Lua and my C++ Game Engine

2014-03-09T00:00:00+00:00

If you want to cut to the chase you can download the code here.

This isn’t a very well written blog post, it’s more of a dump of my thoughts as I tested out LuaBridge and didn’t get to trying out OOLua.

Once again I am starting to integrate Lua in to the next generation of my engine. It’s never a completely smooth process. Exactly where to draw the line between C++ and Lua logic is never immedietly clear.

This is the fifth major revision of my engine (that I can remember) and it’s move toward a component based system has made the dividing line more obvious.

Previously I had driven the engine completely from Lua, the main loop essentially being the first script that the engine loaded. This script drove everything. With a component based system and a new custom editor the “data driving” is done by the editor as it packs levels or scenes together.

Bridging the divide

In previous incarnations of my engine integration of Lua and C++ has be done with:

handcrafted lua code
SWIG
tolua++

I never want to handcraft it ever again, it’s just too time consuming and brittle. SWIG, while easy to use and clever, produces glue code that is slower than needs be and very complicated. tolua++ produces better code than SWIG and has some really nice features but seems to be unmaintained and there are some bugs that need to be worked around.

LuaBridge over OOlua

So I turned to the internet and the top two contenders (that don’t require boost) seem to be OOLua and LuaBridge. They both employ a set of C++ templates to wrap your code, exposing it to Lua.

OOLua has some nice tables showing it is the fastest thing in the world which enticed me BUT it requires Premake, Cpp Unit and Google Mock to build it. This really put me off. I just want to wrap my code, not install another build system and unit testing framework. LuaBridge doesn’t require any external tools or libraries.

LuaBrdge has a more pleasant syntax and the underlying template code seems easier to decipher. To expose a simple struct (taken from the OOLua docs)

struct foo
{
  int bar(float);
};

In OOLua you would enter:

OOLUA_CLASS_NO_BASES(foo)
    OOLUA_NO_TYPEDEFS
    OOLUA_ONLY_DEFAULT_CONSTRUCTOR
    OOLUA_MEM_FUNC_1(int/*return type*/
                   ,bar/*function name*/
                   ,float)/*parameter type*/
OOLUA_CLASS_END

while in LuaBridge:

getGlobalNamespace (L)
    .beginNamespace ("vox")
    .beginClass<foo>("foo")
        .addFunction("bar", &foo::bar)
    .endClass();

LuaBridge seems more concise, less brittle (it uses C++ templates to infer the paramter types and return types) and quite frankly who wants CAPITALS ALL THROUGH THEIR CODE?

Both seem like very well written libraries though so it’s really a matter of personal preference as opposed to some major technical flaw in OOlua.

So I decided to test LuaBridge first. If I could get what I wanted from it then I’d move forward.

Overview of Requirements.

The scripts that I want to attach to my GameObjects are more like behaviours than massive all emcompassing AI scripts. This design is something I learned to like while playing with Unity3D a few months ago. While it adds a certain amount of managment overhead on the engine side it allows a lot of the day to day scripting of a game to be built from small scripts. For example you can write a small script to look at the player and attach it to every enemies head, add a parameter to limit it to when the player is within n meters and you’ve got a nice reusable component that anyone (even non programmers) can drop on trees, pickups, bosses, chickens, anything you want.

I want my scripts to be able to be updated

every frame
before physics
after physics
on a trigger
on a collisions

Or any combination of these. So I need a mechanism where the script lets the engine know how to use the script.

Prototype Interface

Rather then screw up my codebase I created a test project to play with LuaBridge. You can get the code here

All of the code is in the one cpp file main.cpp. test.lua is a test script to attach to GameObjects. It depends on class.lua (fairly generic class definition based on the lua wiki example) and utility.lua (old code, probably also taken from the lua wiki over the years to manipulate tables).

The code is fairly well commented but here is an overview of how it works.

GameObject is a hollowed out container class for entities in a game. It has a few token members to test exposing them to lua.

LoadScript loads a lua script and executes it. The script is expected to register a global function register_script that is called returning a class definition (lua table). This definition is instanstiated and attached to the game object. Load script does some simple caching to prevent test.lua being run multiple times.

Once numscripts GameObjects are created with test.lua attached the GameObjects are ticked. The GameObject::Tick function looks for a function called tick in it’s lua class and if it’s there calls it.

Once that’s all done, everything is deleted to see if we’re leaking memory.

Thoughts

really like LuaRef it’s a very nice way to program lua.
don’t like the way the scripts have to define a callback function, would prefere to return the class definition during the inital script run. However I don’t want to play with the Lua stack and LuaBridge doesn’t seem to provide a clean way to get the table off of the stack.
In tolua++ you could write and expose a C++ function is such a way as it would appear, in Lua, to return multiple values. This was very nice and Lua-ish (search for Multiple returned values here). I can’t see how to do this in LuaBridge.

Gotchas.

Four Character Literal expansion bit me:

void Tick(float dt){
    LuaRef tickfunc = mLuaClass['tick'];
    if(tickfunc.isFunction()){
        tickfunc(mLuaClass, dt);
    }else{
        std::cout << mName << " tick isn't a function: " << tickfunc.type() << std::endl;
   }
}

tickfun will ALWAYS be nil in because clang will convert ‘tick’ in to an integer (note the single quotes, not double quotes for the string tick). You need to access the table thusly:

LuaRef tickfunc = mLuaClass["tick"];

to get access to the function.

There is a setting in XCode to warn about this conversion but it’s off by default. Using single quotes is a habit I’ve picked up from Python unfortunately.

Can’t use LuaRef in a std:: container

Because the default constructor for a LuaRef takes one parameter (the lua_State that it belongs to) if you try to create a map like:

std::map<std::string, LuaRef>

You will get a compiler error.

Moving from qmake to CMake

2014-01-22T00:00:00+00:00

After working with Qt5.x for a few months and really starting to enjoy the flexibility of qmake and QtCreator I decided to try using CMake to build XCode and MSVC proejcts.

After turning on C++11 a few weeks ago I found that QtCreator debugging became more difficult as it couldn’t show me the contents of any of the std containers (vector, list, map etc) without me manually digging in the the internal structure of the class which is quite tedious.

I was worried about leaving begind qmake which to be honst is one of the better make systems out there but I couldn’t tolerate not being able to easily see the data in my app and there has always been a nagging doubt that qmake isn’t flexible enough to allow me to build an editor and a game engine from the same source code with the minimum of fuss.

There are a lot of posts on the internet about CMake not working with the lastest XCode but, bar a few gripes, those issues seem to have been ironed out with the latest CMake (I’m using CMake 2.8.12.1).

If you’ve never used CMake before it uses a heirachy of CMakeLists.txt files to generate platform specific make file, XCode projects or MSVC Solutions (amongst other targets).

The qmake project I started with used the template subdirs which means it was a folder with sub folders with the actual code in it. This is the most common scenario for a reasonably large Qt project.

Creating the CMakeLists.txt files was a manual process of taking the .pro file and copying across the SOURCEand HEADERS definitions to the add_executable statement within the CMakeLists.txt file. For example the glew library:

cache()
! include( ../common.pri ) {
    error( "Couldn't find the common.pri file!" )
}

DEFINES += GLEW_STATIC

TARGET = glew
TEMPLATE = lib
CONFIG += staticlib

SOURCES += \
    glew.c

HEADERS += \
    glew.h \
    glxew.h \
    wglew.h
unix:!symbian {
    target.path = /usr/lib
    INSTALLS += target
}

OTHER_FILES += \
    readme.txt

is transformed to:

add_definitions(-DGLEW_STATIC)

#disable all warning for third party libs
if(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -w")
endif(${CMAKE_SYSTEM_NAME} MATCHES "Darwin")


add_library(glew
    glew.c
    glew.h
    glxew.h
    wglew.h)

Nice and short and fairly simple to read.

CMake has specific support building and linking Qt5 applications and it makes the process of working with Qt reasonably simple:

if(${CMAKE_SYSTEM_NAME} MATCHES "Windows")
    set(CMAKE_PREFIX_PATH "C:\\Qt\\Qt5.2.0\\5.2.0\\msvc2010_opengl" "C:\\WinDDK\\7600.16385.1\\lib\\win7\\i386")
else(${CMAKE_SYSTEM_NAME} MATCHES "Windows")
    set(CMAKE_PREFIX_PATH "~/Qt5.2.0/5.2.0/clang_64/")
endif(${CMAKE_SYSTEM_NAME} MATCHES "Windows")


# Find includes in corresponding build directories
set(CMAKE_INCLUDE_CURRENT_DIR ON)
# Instruct CMake to run moc automatically when needed.
set(CMAKE_AUTOMOC ON)

# Find the QtWidgets library
find_package(Qt5Widgets)
find_package(Qt5Declarative)
find_package(OpenGL REQUIRED)

add_definitions(-DGLEW_STATIC)


qt5_wrap_ui(UI_HEADERS mainwindow.ui)
qt5_add_resources(UI_RESOURCES bsed.qrc)
add_executable(BSEd
    MACOSX_BUNDLE
    mainwindow.cpp
    ...
    BSEd.icns
    ${UI_HEADERS}
    ${UI_RESOURCES})

set(MACOSX_BUNDLE_ICON_FILE BSEd.icns)
SET_SOURCE_FILES_PROPERTIES(
  BSEd.icns
  PROPERTIES
  MACOSX_PACKAGE_LOCATION Resources
  )

target_link_libraries(BSEd Bullet glew LibPng LibZLib
    Polyvox voxcol voxcore voxdata voxmath voxrend voxscene
    ${OPENGL_LIBRARY}
    ${Qt5Core_QTMAIN_LIBRARIES})

if(${CMAKE_SYSTEM_NAME} MATCHES "Windows")
    set_target_properties(BSEd PROPERTIES LINK_FLAGS_DEBUG "/SUBSYSTEM:WINDOWS")
    set_target_properties(BSEd PROPERTIES LINK_FLAGS_RELEASE "/SUBSYSTEM:WINDOWS")
endif(${CMAKE_SYSTEM_NAME} MATCHES "Windows")

qt5_use_modules(BSEd Widgets OpenGL)

The important parts are:

set CMAKE_PREFIX_PATH to include the path to Qt. Otherwise find_package will fail. It’s probably better to use some sort of environment variable for this but since I work on my own I can hard code things.
set(CMAKE_INCLUDE_CURRENT_DIR ON) so that moc’ed files can be found during the build.
set(CMAKE_AUTOMOC ON) enable the mocing of files with QOBJECT defined in them.
use find_package to find the Qt libraries that the app uses.
tell cmake which UI files you are using. qt5_wrap_ui(UI_HEADERS mainwindow.ui) stores the result in the variable UI_HEADERS
tell cmake which resource files you are compiling. qt5_add_resources(UI_RESOURCES bsed.qrc) stores the result in the variableUI_RESOURCES.
make sure you add UI_HEADERS and UI_RESOURCES to the add_executable command.
tell cmake to link with the Qt libraires in target_link_libraries using {Qt5Core_QTMAIN_LIBRARIES}

On windows I found that the default application type was a console app. I didn’t want a little dos box to appear when I ran my app so I changed the subsystem from CONSOLE to WINDOWS with the following:

if(${CMAKE_SYSTEM_NAME} MATCHES "Windows")
    set_target_properties(BSEd PROPERTIES LINK_FLAGS_DEBUG "/SUBSYSTEM:WINDOWS")
    set_target_properties(BSEd PROPERTIES LINK_FLAGS_RELEASE "/SUBSYSTEM:WINDOWS")
endif(${CMAKE_SYSTEM_NAME} MATCHES "Windows")

All of this took about three hours to figure out and get working on OSX and Windows7.

What I’ve gained:

I get to use the native debugging tools which on OSX and Windows7 are faster than QtCreator.
I get to use Xcode and MSVC for editting code both of which has useful features. Especially Xcodes real time error detection. It’s nice to be told your code is fail before you’ve even finished typing it all in.
I get to see inside std containers on both platforms with C++11 enabled.
I much more powerful build system that can cater for multiple targets and build configurations.
A blog post about something ;)

what I’ve lost:

the same editor/debugger (QtCreator) on all platforms. It’s quite nice to move from one machine to the next and the default key bindings are the same.
precompiled headers. This is a big one. CMake doesn’t support PCH out of the box, there is a plugin for CMake called cotire but I haven’t tried it yet.
the simplicity of qmake. CMake is much more powerful and consequently much more tricky to use. Cmake has a lot of documentation but you’ll need good google-fu to find answers quickly.
If you use QtCreator and qmake it feels like you are inside the Qt club. Using CMake seems more brittle and I am now worried that any upgrade to XCode (in particular) will break the build process.
A few hours of my time.

Hope that helps someone out there.

GLSL: Getting active uniforms

2014-01-03T00:00:00+00:00

When you load a vertex and fragment shader in OpenGL you need to pass in data such as the current viewport transform, camera transform and lighting data. You do this via Uniform variables.

Each of these Uniforms is addressed via a location which in true OpenGL style is a GLuint. It’s pretty slow to get the location of a Uniform so it’s advisable to cache the location before using the shader in anger. Here is the code to get the location of all ACTIVE uniforms in a compiled and linked shader program:

int numUni = -1;
GLCHECK(glGetProgramiv( mProgram, GL_ACTIVE_UNIFORMS, &numUni ));
for(int i = 0; i < numUni; ++i){

    int namelen =-1, num=-1;
    GLenum type = GL_ZERO;
    char name[256]; //assume no variable names longer than 256

    /* Get the name of the ith Uniform */
    glGetActiveUniform(mProgram,
                       static_cast<GLuint>(i),
                       sizeof(name)-1,
                       &namelen,
                       &num,
                       &type,
                       name));
    name[namelen] = 0;

    /* Get the location of the named uniform */
    GLuint location = glGetUniformLocation( mProgram, name );

    /* cache the location of the Uniforms in a std::map
    mUniformMap[name] = location;
}

The docs for glGetActiveUniform document the types returned in type but in general it’s not useful information during runtime.

Note this only finds active uniforms so if you have a uniform declared but you don’t use it it’s stripped by the shader compiler and will not appear.

QMAKE_BUNDLE_DATA doesn’t bundle frameworks

2013-12-16T00:00:00+00:00

I’ve been playing with building an App using qmake as the build system. I started with a simple QT App built using the wizard in QTCreator and tweaking it.

I ran in to the problem that I needed to copy the SFML frameworks in to the App Bundle. The QT docs say to use the variable QMAKE_BUNDLE_DATA to trigger the copy but I couldn’t get it to work in QT5.2.

mac{
    PRIVATE_FRAMEWORKS.files = /Users/dazza/work/SFML-2.1-osx-clang-universal/Frameworks/sfml-audio.framework
    PRIVATE_FRAMEWORKS.path = Contents/Frameworks
    QMAKE_BUNDLE_DATA += PRIVATE_FRAMEWORKS
}

No matter what I tried I couldn’t trigger the copying of the framework.

The work around I came up with involves adding a post link step to copy the frameworks using ditto

mac{
    BUNDLE_DIR = $$OUT_PWD/$$TARGET$$quote(.app)/Contents
    QMAKE_POST_LINK += echo Copy Frameworks; \
                   mkdir -p $$BUNDLE_DIR/Frameworks;\
                   ditto \"/Users/dazza/work/SFML-2.1-osx-clang-universal/Frameworks/\" \"$$BUNDLE_DIR/Frameworks\";\
                   ditto \"/Users/dazza/work/SFML-2.1-osx-clang-universal/extlibs/\" \"$$BUNDLE_DIR/Frameworks\";
}

The QMAKE_POST_LINK variable has a script to

echo that it’s copying the frameworks across.
make sure there is a framework directory in the App Bundle.
Copy the SFML frameworks across.
Copy the SFML external dependencies across.

Note I know there is a horrible hardcoded path in there that shouldn’t be there but this is just for testing!

Python Script to build a Texture Page or Sprite Sheet

2013-04-12T00:00:00+01:00

Why pack textures/images?

If you’re making a game then it’s more efficient to tell the hardware:

use packed texture 1
draw primitive 1,2,3,4,5,6
use packed texture 2
draw primitive 7,8,9,10

than to

use texture 1
draw primitive 1
use texture 2
draw primitive 2
...

The less you do, the more efficient rendering is.

If you’re making a website then again it’s quicker to ask the client browser to load one large image (often called a sprite sheet) than to ask it to load many more smaller textures.

Algorithm

The algorithm we are going to use in called Bin Packing. We want to pack the textures in to one larger texture with the minimum of wasted space. Note that there will always be some wasted space, this is an NP complete problem and Bin Packing just gives you a very good solution not a perfect one

2D bin packing works by using the source images to subdivide the destination image creating a tree of ‘used’ areas of the destination image. This sounds a little complication but it’s actually pretty simple.

Code Overview

class BinPackNode

BinPackNode does the work of subdividing space and recording the tree structure of subdivisions creating left and right children. The BinPackNode has:

area which represent the sub rectangle of this image this node covers
leftchild, rightchild if this node has been subdivided leftchild and rightchild are BinPackNodes representing the split of this node.
1
leftchild.area + rightchild.area == self.area
filled is a boolean indicating that this node has been filled with an image and can’t be subdivided.

def pack_images

pack_images is a function that does the grunt work of trawling through the images, sorting them in to size order (largest to smallest) and the iteratively adding them to the BinPackNode tree.

class Rect

Rect is a utility class that represents a rectangle. I used this just to make the logic within BinPackNode.insert more concise and easier to follow.

How does it work

The psuedocode for pack_images is:

sort the images in to size order
create a root BinPackNode with an initial size of the target image
for each image insert it to the root BinPackNode

The pseudocode for BinPackNode.insert is:

if the node has children then
    ask the left child to insert the new image
    if the left child cant fit the image in then
        ask the right child
    if the right child can't fit the image in then
        we can't find the image in to this node
    return

if this node is filled or the new image is bigger than this node then
    we can't fit the image in this node
    return

if the image is the size of this node then
    mark this node as filled
    return

split this node in to a left and right node based on the image
ask the left node to insert the new image
return

Usage

The python code can be run from the command line to pack a set of PNGs that are in a directory:

usage: txtrpacker.py [-h] [-v] [-pad PAD] [-sort SORT] [-maxdim MAXDIM]
                 [--log LOG]
                 src dst

A utility to take a set of png images and pack them in to a power of two image
with pading. The placements of the source images is printed to stdout in the
format: "filename x y x2 y2"

positional arguments:
  src             src directory
  dst             dest png file

optional arguments:
  -h, --help      show this help message and exit
  -v              enable verbose mode
  -pad PAD        padding on each side of the texture (default: 2)
  -sort SORT      sort algorithm one of maxheight,maxwidth,maxarea (default:
                  maxarea)
  -maxdim MAXDIM  maximum texture size permissable.
  --log LOG       Logging level (INFO, DEBUG, WARN) (default: INFO)

using the example data you could enter:

./txtrpacker.py -pad 4 ./testart output.png

Code

This code can be downloaded from this gzip along with some test art from our iOS game Max Astro and our upcoming game (which isn’t public yet). I’ve used to code for a few years so I am pretty confident that it works.

rect.py

__all__ = ['Rect']


class Rect(object):
    """Represent a rectangle in the BinPack tree."""
    def __init__(self, x1, y1, x2, y2):
        self.x1 = x1
        self.y1 = y1  # bottom
        self.x2 = x2
        self.y2 = y2  # top

    def get_width(self):
        return abs(self.x2 - self.x1)

    def set_width(self, w):
        self.x2 = self.x1 + w

    def get_height(self):
        return abs(self.y2 - self.y1)

    def set_height(self, h):
        self.y2 = self.y1 + h

    def get_left(self):
        return self.x1

    def set_left(self, l):
        w = self.get_width()
        self.x1 = l
        self.x2 = l + w

    def get_top(self):
        return self.y2

    def set_top(self, t):
        h = self.get_height()
        self.y2 = t
        self.y1 = t - h

    def get_right(self):
        return self.x2

    def get_bottom(self):
        return self.y1

    def set_bottom(self, y1):
        h = self.get_height()
        self.y1 = y1
        self.y2 = self.y1 + h

    def offset(self, x, y):
        self.left = self.left + x
        self.top = self.top + y
        return self

    def inset(self, d):
        """return a rect which is this rect inset by d in each direction"""
        return Rect(self.x1 + d, self.y1 + d,
                    self.x2 - d, self.y2 - d)

    def inside(self, r):
        """return true if this rectangle is inside r"""
        return self.x1 >= r.x1 and self.x2 &lt;= r.x2\
               and self.y1 >= r.y1 and self.y2 &lt;= r.y2

    width = property(fget=get_width, fset=set_width)
    height = property(fget=get_height, fset=set_height)
    left = property(fget=get_left, fset=set_left)
    top = property(fget=get_top, fset=set_top)
    right = property(fget=get_right)
    bottom = property(fget=get_bottom, fset=set_bottom)

    def __str__(self):
        return "[%f, %f, %f, %f]" % (self.x1, self.y1, self.x2, self.y2)

    def __repr__(self):
        return "Rect[%s]" % str(self)

txtrpacker.py

#!/usr/bin/env python
import argparse
import Image
import logging
from copy import copy
from os.path import join
from glob import glob

from rect import Rect

log = logging.getLogger(__name__)


class BinPackNode(object):
    """A Node in a tree of recursively smaller areas within which images can be placed."""
    def __init__(self, area):
        """Create a binpack node
        @param area a Rect describing the area the node covers in texture coorinates
        """
        #the area that I take up in the image.
        self.area = area
        # if I've been subdivided then I always have a left/right child
        self.leftchild = None
        self.rightchild = None
        #I'm a leaf node and an image would be placed here, I can't be suddivided.
        self.filled = False

    def __repr__(self):
        return "<%s %s>" % (self.__class__.__name__, str(self.area))

    def insert(self, newarea):
        """Insert the newarea in to my area.
        @param newarea a Rect to insert in to me by subdividing myself up
        @return the area filled or None if the newarea couldn't be accommodated within this
            node tree
        """
        #if I've been subdivided already then get my child trees to insert the image.
        if self.leftchild and self.rightchild:
            return self.leftchild.insert(newarea) or self.rightchild.insert(newarea)

        #If my area has been used (filled) or the area requested is bigger then my
        # area return None. I can't help you.
        if self.filled or newarea.width > self.area.width or newarea.height > self.area.height:
            return None

        #if the image fits exactly in me then yep, the are has been filled
        if self.area.width == newarea.width and self.area.height == newarea.height:
            self.filled = True
            return self.area

        #I am going to subdivide myself, copy my area in to the two children
        # and then massage them to be useful sizes for placing the newarea.
        leftarea = copy(self.area)
        rightarea = copy(self.area)

        widthdifference = self.area.width - newarea.width
        heightdifference = self.area.height - newarea.height

        if widthdifference > heightdifference:
            leftarea.width = newarea.width
            rightarea.left = rightarea.left + newarea.width
            rightarea.width = rightarea.width - newarea.width
        else:
            leftarea.height = newarea.height
            rightarea.top = rightarea.top + newarea.height
            rightarea.height = rightarea.height - newarea.height

        #create my children and then insert it in to the left child which
        #was carefully crafted about to fit in one dimension.
        self.leftchild = BinPackNode(leftarea)
        self.rightchild = BinPackNode(rightarea)
        return self.leftchild.insert(newarea)


def _imagesize(i):
    return i.size[0] * i.size[1]

#table of heuristics to sort the list of images by before placing
# them in the BinPack Tree NOTE that they are compared backwards
# as we want to go from big to small (r2->r1 as opposed to r1->r2)
sort_heuristics = {
    "maxarea": lambda r1, r2:  cmp(_imagesize(r2[1]), _imagesize(r1[1])),
    "maxwidth": lambda r1, r2: cmp(r2[1].size[0], r1[1].size[0]),
    "maxheight": lambda r1, r2: cmp(r2[1].size[1], r1[1].size[1]),
}


def pack_images(imagelist, padding, sort, maxdim, dstfilename):
    """pack the images in image list in to a pow2 PNg file
    @param imagelist iterable of tuples (image name, image)
    @param padding padding to be applied to all sides of the image
    @param dstfilename the filename to save the packed image to.
    @return a list of ( rect, name, image) tuples describing where the images were placed.
    """

    log.debug("unsorted order:")
    for name, image in imagelist:
        log.debug("\t%s %dx%d" % (name, image.size[0], image.size[1]))

    #sort the images based on the heuristic passed in
    images = sorted(imagelist, cmp=sort_heuristics[sort])

    log.debug("sorted order:")
    for name, image in images:
        log.debug("\t%s %dx%d" % (name, image.size[0], image.size[1]))

    #the start dimension of the target image. this grows
    # by doubling to accomodate the images. Should start
    # on a power of two otherwise it wont end on a power
    # of two. Could possibly start this on the first pow2
    # above the largest image but this works.
    targetdim_x = 64
    targetdim_y = 64
    placement = []
    while True:
        try:
            placement = []
            tree = BinPackNode(Rect(0, 0, targetdim_x, targetdim_y))

            #insert each image into the BinPackNode area. If an image fails to insert
            # we start again with a slightly bigger target size.
            for name, img in images:
                imsize = img.size
                r = Rect(0, 0, imsize[0] + padding * 2, imsize[1] + padding * 2)
                uv = tree.insert(r)
                if uv is None:
                    #the tree couldn't accomodate the area, we'll need to start again.
                    raise ValueError('Pack size too small.')
                uv = uv.inset(padding)
                placement.append((uv, name, img))

            #if we get here we've found a place for all the images so
            # break from the while True loop
            break
        except ValueError:
            log.debug("Taget Dim [%dx%d] too small" % (targetdim_x, targetdim_y))
            if targetdim_x == targetdim_y:
                targetdim_x = targetdim_x * 2
                if targetdim_x > maxdim:
                    raise Exception("Too many textures to pack in to max texture size %dx%d\n" % (maxdim, maxdim))
            else:
                targetdim_y = targetdim_x

    #save the images to the target file packed
    log.info("Packing %d images in to %dx%d" % (len(imagelist), targetdim_x, targetdim_y))
    image = Image.new("RGBA", (targetdim_x, targetdim_y))
    for uv, name, img in placement:
        image.paste(img, (uv.x1, uv.y1))
    #image.show()
    image.save(dstfilename, "PNG")

    return placement

if __name__ == "__main__":
    _description = """A utility to take a set of png images and pack them in to
    a power of two image with padding. The placements of the source images is
    printed to stdout in the format: "filename x y x2 y2"
    """

    _epilog = " example: txtrpacker.py hud/images data/hud/texturepage.png"

    parser = argparse.ArgumentParser(description=_description, epilog=_epilog)
    parser.add_argument("-v", action="store_true",
                        help="enable verbose mode",
                        default=False)
    parser.add_argument("-pad", type=int,
                        help="padding on each side of the texture (default: 2)",
                        default=2)
    parser.add_argument("-sort", type=str, default="maxarea",
                        help="sort algorithm one of %s (default: maxarea)" % ",".join(sort_heuristics.keys()))
    parser.add_argument("-maxdim", type=int, default=4096,
                        help="maximum texture size permissable.")
    parser.add_argument("--log", type=str,
                        help="Logging level (INFO, DEBUG, WARN) (default: INFO)",
                        default="INFO")
    parser.add_argument("src", type=str, help="src directory")
    parser.add_argument("dst", type=str, help="dest png file")

    args = parser.parse_args()
    numeric_level = getattr(logging, args.log.upper(), None)
    if not isinstance(numeric_level, int):
        log.error('Invalid log level: %s' % args.log)
        exit(-1)
    logging.basicConfig(level=numeric_level)

    if args.sort not in sort_heuristics:
        log.error("Uknown sort parameter '%s'" % args.sort)
        exit(-1)

    #get a list of PNG files in the current directory
    names = glob(join(args.src, "*.png"))
    #create a list of PIL Image objects, sorted by size
    images = [(name, Image.open(name)) for name in names]

    placements = pack_images(images, args.pad, args.sort, args.maxdim, args.dst)
    for area, name, im in placements:
        print("%s %d %d %d %d" % (name, area.x1, area.y1, area.x2, area.y2))

HP ProLiant MicroServer: thoughts

2013-03-03T00:00:00+00:00

I’ve been looking for a new home server for a few months. Recently (the last six years) have been running EPIA Mini-itx motherboards in Cubid cases. They are basically silent and powerful enough to run a small fileserver and webserver.

My EPIA boards were starting to show their age. My longest standing board started to emit a high pitched squeel and sure enough it was a sign of an impending capacitor failure. I swapped it out for an identical replacement and within a few months that motherboard started to show similar problems. I now suspect that the power supply is surging, or in some way, ‘breaking’ my motherboards.

So I looked around and noticed that HP ProLiant Microservers were back on special offer (£100 cash back from HP if you meet their requirements).

The specs looked good but I was dubious about it being quiet enough for my home office. I decided to take the plunge mainly because I could buy the N40L for £209 (with £100 cashback so net cost of £109) from ebuyer.com.

Here are my brief notes on running it as a FreeBSD9.x server:

About 38dB which is quiet enough to have in the next room but too noisy to sleep next to.
Lights on front very bright.
Really well built. It’s not enterprise quality but good enough to sit in the corner of an office.
Excellent access to drives. The four drive caddies are slightly cheap plastic but once they have a drive they are sturdy enough. Since they are not hot swappable they don’t need to be rock solid anyway.
Access to the Motherboard is restrictive due to cabling. You need to unclip a lot of cables from the case and unscrew two thumb screws before you can slide the monthboard out. While this isn’t done too often it’s not for the faint hearted.
Non standard power supply. Might be hard to source a new one.
I wanted to mount a 2.5” SSD drive but it didn’t fit in the caddy. I ended up taping it in to the slot where a DVD drive would go. The motherboard has an extra SATA connector for a DVD drive. This runs at half speed and requires a bios flash to enable full SATA speed.
The motherboard has an exposed USB slot so it can be booted from a USB stick. The case design leaves space for a USB stick to be permenantly mounted in it.
Runs FreeBSD very well. USB, ACPI, temperature sensors (amdtemp module) work. Broadcom network interface works well although only tested on my 100mb network.
Even without the cash back deal this is a very good value server for a small office.
I am running the i386 version of FreeBSD, I really should run the AMD64 but at the moment the box isn’t too loaded and it was easier to just plug in the drive the the previous server, start it up, change a few /etc/rc.conf entries and be done with it.

Basically, when this machine comes back on special offer at ebuyer.com I will be buying another one.

Output from `dmesg`

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-RELEASE #0: Tue Jan  3 07:15:25 UTC 2012
    root@obrian.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386
CPU: AMD Turion(tm) II Neo N40L Dual-Core Processor (1497.55-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x100f63  Family = 10  Model = 6  Stepping = 3
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x837ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,NodeId>
  TSC: P-state invariant
real memory  = 2147483648 (2048 MB)
avail memory = 2020139008 (1926 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <HP     ProLiant>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <HP ProLiant> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of ffb80000, 80000 (3) failed
acpi0: reservation of fec10000, 20 (3) failed
acpi0: reservation of fed80000, 1000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 7bf00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xf8000000-0xfbffffff,0xfe8f0000-0xfe8fffff,0xfe700000-0xfe7fffff irq 18 at device 5.0 on pci1
pcib2: <ACPI PCI-PCI bridge> irq 18 at device 6.0 on pci0
pci2: <ACPI PCI bus> on pcib2
bge0: <HP NC107i PCIe Gigabit Server Adapter, ASIC rev. 0x5784100> mem 0xfe9f0000-0xfe9fffff irq 18 at device 0.0 on pci2
bge0: CHIP ID 0x05784100; ASIC REV 0x5784; CHIP REV 0x57841; PCI-E
miibus0: <MII bus> on bge0
brgphy0: <BCM5784 10/100/1000baseT PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: xx:xx:xx:xx:xx:xx
ahci0: <ATI IXP700 AHCI SATA controller> port 0xd000-0xd007,0xc000-0xc003,0xb000-0xb007,0xa000-0xa003,0x9000-0x900f mem 0xfe6ffc00-0xfe6fffff irq 19 at device 17.0 on pci0
ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ohci0: <OHCI (generic) USB controller> mem 0xfe6fe000-0xfe6fefff irq 18 at device 18.0 on pci0
usbus0: <OHCI (generic) USB controller> on ohci0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe6ff800-0xfe6ff8ff irq 17 at device 18.2 on pci0
usbus1: EHCI version 1.0
usbus1: <EHCI (generic) USB 2.0 controller> on ehci0
ohci1: <OHCI (generic) USB controller> mem 0xfe6fd000-0xfe6fdfff irq 18 at device 19.0 on pci0
usbus2: <OHCI (generic) USB controller> on ohci1
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe6ff400-0xfe6ff4ff irq 17 at device 19.2 on pci0
usbus3: EHCI version 1.0
usbus3: <EHCI (generic) USB 2.0 controller> on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib3: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci3: <ACPI PCI bus> on pcib3
ohci2: <OHCI (generic) USB controller> mem 0xfe6fc000-0xfe6fcfff irq 18 at device 22.0 on pci0
usbus4: <OHCI (generic) USB controller> on ohci2
ehci2: <EHCI (generic) USB 2.0 controller> mem 0xfe6ff000-0xfe6ff0ff irq 17 at device 22.2 on pci0
usbus5: EHCI version 1.0
usbus5: <EHCI (generic) USB 2.0 controller> on ehci2
acpi_button0: <Power Button> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 450
pmtimer0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
ata1 at port 0x170-0x177,0x376 irq 15 on isa0
ppc0: parallel port not found.
acpi_throttle0: <ACPI CPU Throttling> on cpu0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
ugen0.1: <ATI> at usbus0
uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <ATI> at usbus1
uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <ATI> at usbus2
uhub2: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <ATI> at usbus3
uhub3: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <ATI> at usbus4
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <ATI> at usbus5
uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ada0 at ahcich5 bus 0 scbus5 target 0 lun 0
ada0: <V4-CT032V4SSD2 S5FAMM22> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 30533MB (62533296 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad14
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 11699573 Hz quality 800

Output from pciconf -lv

hostb0@pci0:0:0:0:  class=0x060000 card=0x1609103c chip=0x96011022 rev=0x00 hdr=0x00
    vendor     = 'Advanced Micro Devices [AMD]'
    device     = 'RS880 Host Bridge'
    class      = bridge
    subclass   = HOST-PCI
pcib1@pci0:0:1:0:   class=0x060400 card=0x1609103c chip=0x9602103c rev=0x00 hdr=0x01
    vendor     = 'Hewlett-Packard Company'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:6:0:   class=0x060400 card=0x1609103c chip=0x96061022 rev=0x00 hdr=0x01
    vendor     = 'Advanced Micro Devices [AMD]'
    device     = 'RS780 PCI to PCI bridge (PCIE port 2)'
    class      = bridge
    subclass   = PCI-PCI
ahci0@pci0:0:17:0:  class=0x010601 card=0x1609103c chip=0x43911002 rev=0x40 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]'
    class      = mass storage
    subclass   = SATA
ohci0@pci0:0:18:0:  class=0x0c0310 card=0x1609103c chip=0x43971002 rev=0x00 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'SB7x0/SB8x0/SB9x0 USB OHCI0 Controller'
    class      = serial bus
    subclass   = USB
ehci0@pci0:0:18:2:  class=0x0c0320 card=0x1609103c chip=0x43961002 rev=0x00 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'SB7x0/SB8x0/SB9x0 USB EHCI Controller'
    class      = serial bus
    subclass   = USB
ohci1@pci0:0:19:0:  class=0x0c0310 card=0x1609103c chip=0x43971002 rev=0x00 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'SB7x0/SB8x0/SB9x0 USB OHCI0 Controller'
    class      = serial bus
    subclass   = USB
ehci1@pci0:0:19:2:  class=0x0c0320 card=0x1609103c chip=0x43961002 rev=0x00 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'SB7x0/SB8x0/SB9x0 USB EHCI Controller'
    class      = serial bus
    subclass   = USB
none0@pci0:0:20:0:  class=0x0c0500 card=0x00000000 chip=0x43851002 rev=0x42 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'SBx00 SMBus Controller'
    class      = serial bus
    subclass   = SMBus
isab0@pci0:0:20:3:  class=0x060100 card=0x1609103c chip=0x439d1002 rev=0x40 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'SB7x0/SB8x0/SB9x0 LPC host controller'
    class      = bridge
    subclass   = PCI-ISA
pcib3@pci0:0:20:4:  class=0x060401 card=0x00000000 chip=0x43841002 rev=0x40 hdr=0x01
    vendor     = 'ATI Technologies Inc'
    device     = 'SBx00 PCI to PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
ohci2@pci0:0:22:0:  class=0x0c0310 card=0x1609103c chip=0x43971002 rev=0x00 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'SB7x0/SB8x0/SB9x0 USB OHCI0 Controller'
    class      = serial bus
    subclass   = USB
ehci2@pci0:0:22:2:  class=0x0c0320 card=0x1609103c chip=0x43961002 rev=0x00 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'SB7x0/SB8x0/SB9x0 USB EHCI Controller'
    class      = serial bus
    subclass   = USB
hostb1@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x12001022 rev=0x00 hdr=0x00
    vendor     = 'Advanced Micro Devices [AMD]'
    device     = 'Family 10h Processor HyperTransport Configuration'
    class      = bridge
    subclass   = HOST-PCI
hostb2@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x12011022 rev=0x00 hdr=0x00
    vendor     = 'Advanced Micro Devices [AMD]'
    device     = 'Family 10h Processor Address Map'
    class      = bridge
    subclass   = HOST-PCI
hostb3@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x12021022 rev=0x00 hdr=0x00
    vendor     = 'Advanced Micro Devices [AMD]'
    device     = 'Family 10h Processor DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
hostb4@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x12031022 rev=0x00 hdr=0x00
    vendor     = 'Advanced Micro Devices [AMD]'
    device     = 'Family 10h Processor Miscellaneous Control'
    class      = bridge
    subclass   = HOST-PCI
hostb5@pci0:0:24:4: class=0x060000 card=0x00000000 chip=0x12041022 rev=0x00 hdr=0x00
    vendor     = 'Advanced Micro Devices [AMD]'
    device     = 'Family 10h Processor Link Control'
    class      = bridge
    subclass   = HOST-PCI
vgapci0@pci0:1:5:0: class=0x030000 card=0x1609103c chip=0x97121002 rev=0x00 hdr=0x00
    vendor     = 'ATI Technologies Inc'
    device     = 'M880G [Mobility Radeon HD 4200]'
    class      = display
    subclass   = VGA
bge0@pci0:2:0:0:    class=0x020000 card=0x705d103c chip=0x165b14e4 rev=0x10 hdr=0x00
    vendor     = 'Broadcom Corporation'
    device     = 'NetXtreme BCM5723 Gigabit Ethernet PCIe'
    class      = network
    subclass   = ethernet

Hope this helps someone,

be seeing you.

Graphing Electricity Usage with Current Cost and Munin

2013-02-26T00:00:00+00:00

Many years ago I bought a Current Cost meter. Infact it was so long ago now that they have called my version The Classic. I wanted to see how much electrical energy all my servers, home computers and gadgets were using.

The Current Cost package contains two main parts, a transmitter and the display. The transmitter is hooked around your house/offices main power in BEFORE the consumer meter. The consumer meter is the big box that is usually under your stairs. The meter constantly sends power usage to the display which shows the watts currently being used, watts over time and, with a little bit of input from the user how much your energy usage is going to cost you over time.

The reason I went with a Current Cost meter is becuase it has a serial connection on the bottom of the display unit. One annoyance is that it uses a RJ45 connector so you either need to stump up for a custom cable from Current Cost or make your own (google it, there are a number of sites with pinouts and instructions).

I already monitor pretty much every aspect of my servers using Munin so it seemed like a natural solution for graphing the Current Cost data over time.

Sending data to Munin is fairly easy and it has a simple plugin interface. There are some complexities to it as Munin uses RRDTool to render it’s graphs and the options for RRDTool are complex and confusing.

The results are graphs like this:

showing watts used over time

showing temparature at or near the Current Cost display

OK they are not very pretty but that’s not the point. I can see that right now as I am typing this my house is consuming about 440 watts. At this rate, per hour I am using 440Wh (watt hours) or 0.44 kWh which is (on average) 0.44 * £0.15 ~ £0.06 per hour

Now as you can see, through the day we get massive spikes, they are usually boiling water for coffee or cooking.

The most interesting thing for me is that having the data before me modifies my behaviour. For example I finally unplugged my Wii today, 4 months after getting my WiiU. It’s been there quietly sucking up a few kWh per day. It hasn’t cost me a lot but as they always say:

If we all did it we’d save the planet!

That is a direct quote from Al Gore!

So here is the python code that you can use as a munin plugin

#!/usr/bin/env python
# Script to extract data from a CurrentCost meter over the serial line
# tested on FreeBSD 9.0 2013/1/22
# dazza

import sys
from datetime import datetime
import xml.etree.ElementTree as et


class DataBlock(object):
    """class to encapsulate the inforkation from a blob of data from the
    currentcost box"""
    def __init__(self, elemtree):
        ft = elemtree.findtext

        self.dsb = int(ft("date/dsb"))
        today = datetime.today()
        self.date = datetime(today.year, today.month, today.day,
                             hour=int(ft("date/hr")),
                             minute=int(ft("date/min")),
                             second=int(ft("date/sec")))
        self.src = ft("src/name")
        self.src_id = int(ft("src/id"))
        self.src_type = ft("src/type")
        self.source_ver = ft("src/sver")
        self.temp = float(ft("tmpr"))
        self.watts = []
        for idx, channelid in enumerate([1, 2, 3]):
            self.watts.append(float(ft("ch%d/watts" % channelid)))

    def __str__(self):
        return "DSB: %(dsb)s\nDATE: %(date)s\nSRC: %(src)s\nSRCID: %(src_id)s\nSRC_TYPE: %(src_type)s\nSOURCE_VER: %(source_ver)s\nTEMP: %(temp)s\nCH: %(watts)s" % self.__dict__


def _readvalues():
    """read a blob of data from the serial connection to the CurrentCost"""
    f = open("/dev/cuaU0")
    db = None
    while True:
        msgstr = f.readline()
        if "<msg>" in msgstr and "</msg>" in msgstr:
            try:
                msg = msgstr[msgstr.find("<msg"):msgstr.find("</msg>") + 6]
                elemtree = et.XML(msg)
                db = DataBlock(elemtree)
                break
                msgstr = msgstr[msgstr.find("</msg>") + 7:]
                break
            except Exception, e:
                print "Exception: " + str(e)
                msgstr = ""
                pass
    f.close()
    return db


def _config_watts():
    """set the configuration for the watts graph"""
    print "graph_title Energy Usage (watts)"
    print "graph_vlabel Watts"
    print "watts_1.label Watts Ch. 1"
    print "watts_2.label Watts Ch. 2"
    print "watts_3.label Watts Ch. 3"
    print "graph_args -l 0"
    print "graph_scale no"
    print "graph_category Home"


def _config_temp():
    """set the config for the temperature graph"""
    print "graph_title Temperature (C)"
    print "graph_vlabel C"
    print "temp.label Temp"
    print "graph_args -l 0"
    print "graph_scale no"
    print "graph_category Home"


def _watts():
    """emit the values fo the watts graph"""
    db = _readvalues()
    print "watts_1.value", db.watts[0]
    print "watts_2.value", db.watts[1]
    print "watts_3.value", db.watts[2]


def _temp():
    """emit the values for the temperature graph"""
    db = _readvalues()
    print "temp.value", db.temp

command = sys.argv[1] if len(sys.argv) > 1 else ""
graphtype = sys.argv[0].split('_')[1]

if command == "config":
    if graphtype == "watts":
        _config_watts()
    elif graphtype == "temperature":
        _config_temp()
    else:
        raise Exception("Unknown graph type %s" % graphtype)
else:
    if graphtype == "watts":
        _watts()
    elif graphtype == "temperature":
        _temp()
    else:
        raise Exception("Unknown graph type %s" % graphtype)

There is a lot going on here so I’ll summerise the code:

Munin plugins are usually symbolic links to scripts. This script gathers either the watts being used or the room temperature depending on the name of the symbolic link. So if the link is called currentcost_watts it will return information about watts, if it’s currentcost_temperature it will return temperature information. This is normal Munin pratice. (see line 88)
Munin passes in the argument “config” if it wants the plugin to output the RRDGraph config for the data. (see line 87)
The _readvalues() function is hardcoded to read from FreeBSD’s serial device /dev/cuaU0 your Current Cost meter might appear somewhere else.
The _readvalues() function returns a DataBlock class. Reading from serial means you get partial data at times so the code iterates over the serial stream until it’s get all the data needed for ONE DataBlock. Because the munin script is being called once every 5 minutes it doesn’t matter too much.
Where there is a lot of power being used I seem to get values appearing on Channel 2. The Current Cost meter I have can read from three meters. I only have one meter so I don’t know how this is happening. I assume it’s a bug in the Current Cost meter but it could be in my code.

I hope this helps someone, good luck getting your energy bills down. I wish there was a similar bit of hardware to measure natural gas usage.

be seeing you.

Using Python and Tkinter to capture script output

2012-10-26T00:00:00+01:00

In order to build the assets for my games I have a series of python scripts that know how to take files TexturePacker, Tiled, and Blender and build them for use within the engine. My main platform is OS X and it has good terminal support and more importantly I am comfortable working with the terminal. The artist I work with is not so comfortable and he works on Windows where cmd.exe is painfully outdated and basically useless.

To counter this I wrote a small Python Tkinter script do display a window, text area, some checkboxes and a build button to make it a bit easier for him to work. Crucially it makes it easier to notice build errors and send me bug reports.

There is some tricky code in there to spawn a process and read it’s stdout and stderr in python.

proc = subprocess.Popen(cmdlist,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.STDOUT,
                        universal_newlines=True)

while True:
    line = proc.stdout.readline()
    if not line:
        break
    self.addText(line)
    #this triggers an update of the text area, otherwise it doesn't update
    self.textarea.update_idletasks()

The key here is to spawn the process, redirect both it’s stdout to the pipe between this process and the child process. set it’s stderr to be the same as it’s stdout (so errors appear in the pipe) and then repeatedly read the stdout pipe one line at a time refreshing the textarea when a new line is added.

While this is going on the app is unfortunately unresponsive to the user (as Tk.mainloop() isn’t being called) but it’s still better than cmd.exe especially since the text widget has colored text and an infinite (as in 65KB) buffer.

Colorizing the text turned out to be incredibly easy with Tkinter. When adding a line to a text widget you pass tags to the widget. These tags represent traints that should be applied to the text.

Before using tags they need to be setup:

self.textarea.tag_config("errorstring", foreground="#CC0000")
self.textarea.tag_config("infostring", foreground="#008800")

Now calling

self.textarea.insert(INSERT, "ERROR", ('errorstring',))

inserts text with the errorstring tag resulting in red text appearing. Note that the tags parameter needs to be a tuple (thus the trailing comma after the tag name) and specifically NOT a list (which is a little annoying).

The code is available as a github gist here: It incudes a sample script that randomly prints messages, pauses and then returns after printing n messages.

Here are some screenshots so you can see what it is. They are all basically the same but showing it running on FreeBSD, OS X, Windows.

Hope this helps someone out there.

dazza

Generating Anagrams in Lua

2012-10-22T00:00:00+01:00

I’m working on a game where I need to generate anagrams of words. My new game engine, which is Max Astro’s engine but with a few tweeks, integrates Lua and all the game logic is in Lua.

So given a string like “chicken” I want “kcehcine” or some other random reorganisation of the letters in chicken.

Firstly, accessing the individual letters in a Lua string is not that easy, there is no subscript access so I needed to turn the string in to a table:

local word = "chicken"
local letters = {}
for st in word:gmatch(".") do
    table.insert(letters, st)
end

Now the table letters has all the letters in the word chicken in order. Now I shuffle the table using my table_shuffle function :

function table_shuffle(t)
    local n = #t
    while n >= 2 do
        -- n is now the last pertinent index
        local k = math.random(n)
        -- Quick swap
        t[n], t[k] = t[k], t[n]
        n = n - 1
    end
    return t
end

You can turn the table back in to a string using table.concat.

I’m not sure how efficient this is but it suits my needs, I don’t call it often.

The table shuffle function came from here which is an implementation of the Fischer-Yates shuffle algorithm.

Starter Blender Exporter

2012-07-26T00:00:00+01:00

I’ve settled on using Blender 2.6 both as a level editor and modeller for my next game (code name BBM). BBM is a 2D game but modelling the characters and rendering the sprites will save a lot of time. Blender isn’t the best level editor but it’s pretty close and suits my needs.

The ability to add arbitrary properties to an object in Blender 2.6 makes Blender a “good enough” fit for me. Certainly it’s flexible enough that it’s usable and will save a lot of time over writing my own stand alone or in game editor.

I had written an export plugin for Blender < 2.6 (many years ago) but the whole python interface has changed so I needed to write a new one. I decided to just export the scene to an XML file which I can post process in to a binary file for loading in to the engine. While this adds a step between the editor and the game it does make the process of writing the exporter and understanding the data a lot more simple. Blender doesn’t offer the perfect environment for writing python plugins so keeping them simple is a good idea.

The exporter is pretty simple. I’m releasing the code under a BSD license. It comes in two parts, export26.py which exports an xml file of the scene and load.py which loads the XML file creating a dictionary of the ‘types’ that the exporter exports.

The loader and exporter are a great starting point to a custom exporter of your own. They are pretty generic at the moment and pretty untested!

export26.py load.py

Logical Bit Operators in Lua When Using Floats

2012-02-27T00:00:00+00:00

I’m making some changes to the codebase that drives Max Astro before making my next game. The biggest change is that the engine is going to be controlled by Lua. Lua will run the main logic loop and lua will be the configuration language.

One of the oddities of Lua, at least from a game makers perspective, is that there are no bit logical operators (AND, OR, XOR). The reason for this is that Lua doesn’t have an internal representation of an Integer. Lua, by default uses doubles for all numbers. There are of course a number of add on libraries to get around this.

In luaconf.h there is an option to define LUA_NUMBER as a float and I imagine most game makes do this at least on mobile platforms… and this breaks most bitop libraries for Lua. I say most because it broke the few I tried before just modifying one and moving on.

Making it Work

My library of choice is BitLib which works with Lua 5.0/5.1. Like all the libraries it’s small and easy to embed in your engine code.

Download the source code unpack it and copy bit_limits.h and lbitlib.c to your project.

Open up bit_limits.h and change it to this

#if defined(LUA_NUMBER_FLOAT)
#   define BITLIB_FLOAT_BITS 24
#   define BITLIB_FLOAT_MAX  0x7FFFFF
#   define BITLIB_FLOAT_MIN  (-0x800000L)
#   define BITLIB_FLOAT_UMAX 0xffffff
#elif defined(LUA_NUMBER_DOUBLE)
#   define BITLIB_FLOAT_BITS 53
#   define BITLIB_FLOAT_MAX  0xfffffffffffffL
#   define BITLIB_FLOAT_MIN  (-0x10000000000000L)
#   define BITLIB_FLOAT_UMAX 0x1fffffffffffffUL
#else
#   error("Need to define float or double for numbers")
#endif

If you understand how floats are represented then all I’ve done is twiddle some bits in a calculator to work out the limits of the significand (mantissa to old people like me).

Lua Coders Know Your Limits

The downside of using floats in this case is that for each number you have, you only have 24 bits to toggle on and off. This is the number of bits in the significand (mantissa).

Now you can do

local COLLISION = bit.lshift(1, 0)
local WALL = bit.lshift(1, 1)
local PLAYER = bit.lshift(1, 2)
local CRATE = bit.lshift(1, 3) 
--annoyingly we can't test with truth values here
local flags = 1
assert(bit.band(flags, COLLISION) ~= 0)
assert(bit.band(flags, WALL) == 0) 
flags = bit.bor(COLLISION, CRATE)
assert(bit.band(flags, COLLISION) ~= 0)
assert(bit.band(flags, CRATE) ~= 0)
assert(bit.band(flags, PLAYER) == 0)

Much thanks has to go out to Reuben Thomas for releasing this library under a BSD license allowing me to embed this code without having to read a tome of legalese.

Hope this helps someone out there twiddle their bits.

be seeing you

dazza

iPhone App Version Information In XCode

2011-01-17T00:00:00+00:00

When you create an iOS Application in XCode by default the version of the application is stored in-Info.plist. It would be really handy to get this in to code for compile time checks and #defines.

PList files are actually XML files which makes them easy to read in a shell script but complex to manipulate in bash. Luckily there is a utility called PListBuddy installed in /usr/libexec (not the usual path for command line utilities in OS X) which can manipulate PList files in various ways. For example:

/usr/libexec/PlistBuddy -c "Print :CFBundleVersion" Stranded/Stranded-Info.plist

outputs

0.28

showing that the current development version of stranded is 0.28 (long way to go to get to 1.0)

I wanted to output a header file called version.h from my various targets based upon the value of CFBundleVersion in Stranded-Info.plist. To accomplish this I added a small build step to each target which ran the following script:

PLISTBUDDY='/usr/libexec/PlistBuddy'
PFILE='Stranded-Info.plist'
VERSIONFILE='Classes/version.h'
version=$($PLISTBUDDY -c 'Print :CFBundleVersion' $PFILE)
IFS="." arr=( $version )
echo "#ifndef VERSION_H" > $VERSIONFILE
echo "#define VERSION_H" > $VERSIONFILE
echo "#define MAJOR_VERSION ${arr[0]}" > $VERSIONFILE
echo "#define MINOR_VERSION ${arr[1]}" > $VERSIONFILE 
echo "#endif" > $VERSIONFILE 
echo "" > $VERSIONFILE

This should be placed as a ‘New Run Script Build Phase’ with your info.plist file as input and your output file as the version.h file you want to generate.

It’s important that this build script is the first script to run so that version.h is updated BEFORE the application code is built.

I’ve renamed the build script to “Update Version” so I remember what it is.

Now in code you can write:

#include "version.h"
... 
HUD::Get().SetLabel("versionlabel", StrFormat("v%d.%d", MAJOR_VERSION, MINOR_VERSION));

Metalify - A Spotify Clone

2010-09-05T00:00:00+01:00

For a few months I’ve wanted to make a clone of Spotify but I haven’t been able to justify the time. Last week I decided to take a week off and give it a go. The justification for this is that I wanted to deploy a Django website and learn Qt. I named my project Metalify.

Server Side

Most of the content that you see in the spotify client is actually presented via webpages. So in Metalify I deployed a Django server to server album listings, individual albums with thier track listing and a little blurb. I didn’t get a chance to implement the search features that Spotify has but tbh they are just more webpages.

Client Side

I had a hunch that Spotify was written in Qt (statically linked). I’ve used many many frameworks for apps (MFC, WxWidgets, Fox, Cocoa) but never Qt. Qt suits my needs as I wanted a single code base that compiles for MacOS X and Windows but looks native.

As you can see they look pretty native on each platform.

Having never used Qt before I was surprised at how easy it was to get stuck in. Qt comes with a Qt Designer an interface builder application allowing you to build an interface and test it within the designer. Despite a few oddities it’s pretty easy to use and saves out a .UI file that you can load into you mainwindow class.

Within Qt Designer you can edit style sheets to style your interface. Use

*{ 
    color: rgb(231, 231, 231);
}

QFrame{ 
    border: none; 
    background-color: #939393; 
}

QPushButton{ 
    color: rgb(10, 10, 10); 
}

This is a really good idea. I actually LOVE this feature. It makes so much sense. The downside is that it carries all the baggage of CSS, your CSS file can quickly become a complete mess!

The CSS allowed me to style the play/skip buttons and make them round

Use

#playButton{ 
    width: 32px; 
    height: 32px; 
    background: url(:/resources/play.png); 
    border-radius: 32px; 
}

#playButton:hover{ 
    background: url(:/resources/play_hover.png);
}

#playButton:pressed{ 
    background: url(:/resources/play_depressed.png); 
}

By setting the border-radius for a button you can make it round, change for mouse over and being clicked. I’ve never made a cool looking button in a application framework this easily before.

The downside of the style sheet system is that not all widgets support it. This meant that I couldn’t style the volume (Phonon::VolumeSlider) or scrub widget (Phonon::SeekSlider). As far as I could work out the only way to customize the look of these widets was to subclass and override Widget::paint which seemed like overkill for a week long experiment.

Audio

Qt comes with a audio framework called Phonon. It’s not very mature but is workable. Combined with the network code it actually made playing MP3s on the mac VERY easy. Infact, taking some sample code I was downloading and playing MP3s within an hour.

Spotify has a caching mechanism so you can play tunes offline (if you pay) and to limit their bandwidth costs. I implemented a simple cache system which stored the downloaded MP3s in a platform specific cache folder. The logic is thus: If the file isn’t in the cache start playing the file streamed from the server (so it starts to play within a few seconds) and also download it from the server. This isn’t ideal as the file is downloaded twice. It was however quick to implement and allowed me to use the high level Qt classes.

On Win32 audio playback wasn’t so easy. For some reason certain MP3s just didn’t playback on Win32. Phonon on Win32 wraps DirectShow and DirectShow wraps various decoders. Despite the Qt Capabilities program saying it should be able to decode MP3 it kept failing for specific MP3s. I re-encoded them using Lame (default settings) and then they were able to play. This unnerved me a little and was my first hint that maybe Phonon isn’t as mature as I had hoped.

Integration of the Website and Client

As I said before Spotify displays most of it’s information via webpages. Similarly the right hand pane of Metalify is a QWebView. Integrating the website and the client was really easy.

webView->page()->mainFrame()->evaluateJavaScript(jQuery); 
... 
webView->page()->mainFrame()->addToJavaScriptWindowObject("player", player);

evaluateJavaScript allows me to inject any javascript I wished into the HTML page downloaded from the server. addToJavaScriptWindowObject allowed me to expose a carefully crafted C++ object to the Javascript running in the page.

Using these two access routes I was able to work out the currently selected track, see what tracks to start downloading and update the currently playing track. This was all a lot easier than I had expected.

Cross Platform Dev

The primary reason for choosing Qt was cross platform development. Qt actually helped (as opposed to a lot of tools that claim to help but actually get in the way). I read a few pages of the extensive Qt docs and I was away.

Qt uses a .pro file which is a make file for it’s qmake utility. I used qmake to create an XCode project and then XCode to compile/debug. Moving to WinXP I used QTCreator as an IDE which builds using Ming-w32 (a windows port of gcc). QTCreator is a fairly complete IDE that comes with QT and was fine for my needs. I don’t know if I could use it day in day out as I basically used it to call qmake! I could have chosen to build with Visual Studio but since my main aim was to get the code and build (not write or debug) on Windows it didn’t make sense to complicate the build process too much on Win32. Most of the time Qt hid the differences between Win32 and OS X from me and everything just worked. Very impressive.

Client Installer

As I was making a fake consumer application I needed to make an installer. As I had used Nullsoft Installer System (NSIS) before I quickly created an install/uninstall script and found myself on the usual DLL nightmare of Win32. Qt recommend the use ofDependency Walker which analyses an exe and lists all the DLLs required to run it. Turns out I needed a lot of them:

File "c:\Qt\2010.04\mingw\bin\mingwm10.dll"
File "c:\Qt\2010.04\mingw\bin\libgcc_s_dw2-1.dll"
File "c:\Qt\2010.04\qt\bin\phonon4.dll"
File "c:\Qt\2010.04\qt\bin\QtCore4.dll"
File "c:\Qt\2010.04\qt\bin\QtGui4.dll"
File "c:\Qt\2010.04\qt\bin\QtNetwork4.dll"
File "c:\Qt\2010.04\qt\bin\QtWebKit4.dll"
File "c:\Qt\2010.04\qt\bin\QtXmlPatterns4.dll"
This pushed the installer size to ~10MB.

Making an installer on MacOS was a little bit easier. Qt ships with a tool to look at an app and build a DMG (Disk Image) which the end user can mount and then copy the .App to their Applications folder. The one App contains all the Shared Libraries (DLLs) that the App needs to run.

macdeployqt client.app -dmg -no-plugins

To my horror this made an app that was 96MB and the DMG file was 32 MB. To be fair this is a Universal app so the App and Shared libraries were in x86 and PPC formats but that still over twice the size of the Win32 installer!

Lessons Learned

I really enjoyed writing working on this project and I think I learned and achieved a lot.

Qt REALLY is a great library. The fact that it comes under the free LGPL license or a proprietary license is very generous of Nokia. While I’d rather not see the GPL, I favour more liberal BSD/MIT licenses, I do understand that they want any good changes to Qt to come back to them. After all they are giving you the source to a VERY good library to play with so it’s good to give back.

If Spotify is built using Qt then they must be statically linking with QT. My app is massive compared to theirs. IF they are not useing Qt then I’d love to know why not as it really is ideally suited to this kind of task. It’s easily skinnable, robust, complete and well documented.

Django is very simple to use if a little tedious to set up and deploy behind apache. It’s not that different from TurboGears. It’s well documented and heavily indexed by google. Django Snippets is a great resource for niggly questions like “how to I efficiently sever a multi MB file”.

Qt’s community seems to be a bit lacking. I found few useful resources of creating custom Widgets. This seemed a shame to me.

Issues

The code is most definitely alpha quality. Firstly the download is far too big. I’m pretty sure static linking Qt into the client would make it bigger but not needing the Qt DLLs/Shared Libraries would be a win.

On Mac OSX the client takes 100% CPU time while playing an MP3. I only noticed this after finishing the code (I’ve got 4 CPUs so I didn’t notice loosing one of them). I’m pretty sure this is a Phonon bug/issue as All I do is ask Phonon to play the MP3 and then wait for events to fire.

The code to cache MP3s doesn’t condsider the size of the cache so it will grow for ever. Spotify sets a limit to downloaded content defaulting to 10GB.

If you play an album and then navigate to another album the player get confused. This would be trivial to fix but I’ve left it as an exercise for the reader ;)

The left hand pane of the client has links to What’s New, Radio etc these are all placeholders and do nothing. They would be easy to hook up to urls in the web view though.

The test website has three albums on it (quite a lot less than spotify but it’s close). I haven’t got permission to distribute these releases but they are all freely available from the bands for free as downloads. Please remember this is just sample code and a sample website.

Conclusion

I REALLY enjoyed this project. Cloning someone elses work is a great way to focus on the code and it’s easy to make progress and judge the quality of what you create. I think what is there is a good jumping off point for creating a competing product to spotify.

If you’re interested in taking the code and running with it please do. Execution Unit Ltd is placing this code into the public domain.

Source Code

You can get the source code for the project on BitBucket.

Apple TnC Changes and Lua

2010-08-11T00:00:00+01:00

With the beta release of iPhone OS 4.0 Apple has changed the terms and conditions for developers who want to target the iPhone platform (note that includes the iPod Touch and iPad).

There has been a of discussion about changes to clause 3.3.1 most notably by Gruber on Daring Fireball here and here. Steve Jobs references the second article in a reply here

3.3.1 — Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs. Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

my emphasis added

Gruber basically makes the point that Apple is trying to keep substandard abstraction layers out of the iPhone eco system so that they can control the quality more easily. This is a fair point. The strength of the iPhone eco system is that there is a certain degree of uniformity of the interfaces and styles of interfaces. It should be noted that Apple has always reserved the right to reject apps at submission time for being substandard or not following their interface guidelines.

So why am I writing about this? Well a few weeks ago I finished my Lua abstraction layer and there is a lot of chatter about whether Lua breaks the new TnC. Lua itself is written in C which is good and you embed it in YOUR code and it largely only interfaces with your code bar system calls for malloc, fopen, etc. So Lua (the runtime) compiles and links against the documented API and it’s written in C, no problem there. The other side of the equation is the runtime scripts that my engine runs to perform AI. Now they are compiled but do NOT link directly against Apple API’s. The compiled Lua, in it’s bytecode form only interfaces with the Lua runtime and the runtime (which meets the new TnC) makes the system calls.

So my reading of this is that Lua doesn’t break the TnC.

But I might be wrong and that’s fine, I’ll develop my game using Lua and port across to C++ if the app is rejected. I don’t think it will be but if it is I will just bite the bullet. I’m not saying I will like it, but I will do it.

Having worked in the Video Games industry for 15 years I am not surprised by seemingly draconian Terms and Conditions from hardware manufacturers. Anyone who has ever worked with Sony, Nintendo or Microsoft is accustomed to arcane submission protocols, arbitrary enforcement of rules and seemingly arbitrary rejections upon submission.

The deal we all strike with the platform providers is that they keep the eco system alive and well, they keep up standards and we try our best to comply with those standards. In return we get access to their customers. With Apple this is even more real as there are millions of units in the hands of people who are willing to pay money for a wide variety of games and apps. Apple provides an App Store, International Billing and Distribution network all for a fixed cost per sale. It’s an amazing deal for an indie developer. The reason I think a lot of people are complaining is because they think that the App Store is or should be a democracy. Well if that is what the market place wants then I am sure apple will deliver that but at the moment we’ll just get used to playing by Apple’s rules and getting access to 50,000,000+ users worldwide!

NvStrip and Blender – Stripifying Models for the iPhone/iPad.

2010-06-14T00:00:00+01:00

What Is Stripification?

Stripification is the process of turning lists of triangles in an arbitrary order into as many lists of triangles that contain adjacent edges. There is a great explanation of a triangle strip of wikipedia.

Why would you stripify?

On most platforms Triangle Strips render more quickly. The iPhone and iPad are generally fill bound (meaning the slowest part of the pipleline is plotting pixels) so, for most cases, there doesn’t seem to be a speed up.

Triangle strips do however take up less memory because a triangle strip index takes up less memory.

NVTriStripLib

A long time ago NVidia released source code for striping called the NvTriStrip Library. It has a very simple interface, can generate multiple strips or one long strip with distinct strips joined by degenerate triangles (stitching). It can also limit the length of a strip to maximise caching on the NVidia platform (something I don’t really care about).

One small problem with NvTriStrip Lib is that it doesn’t compile on XCode because of a few small C++ scope issues. These are easy to solve with a few gotchas to look out for.

Blender Integration

If you’ve read my previous blogs you’ll know that I am using blender and a level editor, animation package and basic modeller. My exporter is written in Python so I needed to integrate python and C++. I could have written a python module and imported it into the exporter code but I decided to write a command line utility.

Writing a python module is a little bit more complicated then I wanted. I needed a small wrapper around the NVTriStrip code which took the triangle vertex indices and returned the strip data I needed. Piping the data too and from a command line app seemed the easiest route.

Calling A Command Line Utility From Python

In the new(ish) world of python you use the subprocess module to call a command line up from python and open it’s stdin and stdout pipes (or files).

blender_path = os.path.abspath(os.path.dirname( sys.argv[ 0 ] ))
STRIPY_PATH = os.path.join(blender_path, "Stripy")
p = subprocess.Popen([STRIPY_PATH, "-s"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=True)

subprocess.Popen is passed a list of arguments (the first being the path to the executable application), the -s tells the app to stitch disparate strips together. subprocess.PIPE tells Popen that we would like the application’s stdin and stdout to be pipes.

Once the app is open we can access it’s stdin and stdout via p.stdin and p.stdout. So to send it the triangles vertex indices we do something like:

for i in indices: 
    p.stdin.write("%d " % i) 
    p.stdin.write(" -1\n")

The -1 at the end is an an invalid index for a triangle index and we use it to mark the end of the input stream. We could close the p.stdin but I decided to do this for some reason.

As soon as stripy get a -1 it starts to process the indices that you have passed it printing the results to it’s stdout, so we can read the data via p.stdout.

numstrips = int(p.stdout.readline())
if numstrips > 0:
    raise ExUnitException("Expected at least one strip to be returned")
for sidx in xrange(numstrips):
    striptype = int(p.stdout.readline())
    numindices = int(p.stdout.readline())
    indices = map(int, p.stdout.readline().split())

Stripy

I’ve put up the code for Stripy including the (slightly) modified version of the NVTriStrip code which is owned by NVidia and all credit needs to go to them for putting it out there for us to play with.

You can download it here. The application takes two command line arguments -cs n allowing you to set the hardware cache size (you can ignore this) and -s to stitch disparate strips together.

Have fun,

dazza.

A Pythonic “Center New” in Blender

2010-04-28T00:00:00+01:00

Part of my Art pipeline for Tentacles is to import a directory of 3DS files. Each of these files is a map piece and the map pieces are instanced to create a map. That way I can store the geometry for a shack once but instance it across the level many times with a unique transform (scale, translation and rotation).

Here you can see two map pieces and the big white box is the collision mesh that will be used for the map pieces

Upon importing the 3DS file I join the separate meshes together into one mesh and then reset the center of the model in the XY (ground) plane so that the map pieces can be snapped together and tiled easily.

Center New

To move the center of an object to the center of the mesh Blender has a button calledCenter New (Editing Pane, Mesh Subpanel). Unfortunately there isn’t an easy way to invoke this from within python.

I found this code on the internet quite quickly but I found that for some models it just wasn’t working. Infact for some models if I called the function twice the center changed twice!

I rejigged the code a little to get it to work every time on Blender 2.49. Other then renaming the variables to make it a little easier to read I get the objects bounding box before manipulating the object and it’s meshes matrix. Use

def center_new(bobject, xfactor=1, yfactor=1, zfactor=0):
    bbbox= bobject.getBoundBox()
    bmesh=bobject.getData(0, 1)
    matrix= bobject.getMatrix()
    bmesh.transform(matrix)
    bmesh.update()
    matrix=Blender.Mathutils.Matrix(
                  [1.0,0.0,0.0,0.0],
                  [0.0,1.0,0.0,0.0],
                  [0.0,0.0,1.0,0.0],
                  [0.0,0.0,0.0,1.0])

    bobject.setMatrix(matrix)
    bboxcenter=[bbbox[0][n] + ( bbbox[-2][n] - bbbox[0][n] )/2.0 for n in [0,1,2] ]
    matrix= bobject.getMatrix()

    matrix=Blender.Mathutils.Matrix(matrix[0][:],
                         matrix[1][:],
                         matrix[2][:],
                         [-bboxcenter[0] * xfactor,-bboxcenter[1] * yfactor,
                         -bboxcenter[2] * zfactor,1.0])
    bmesh.transform(matrix)
    bmesh.update()
    matrix=Blender.Mathutils.Matrix(matrix[0][:],
                     matrix[1][:],
                     matrix[2][:],
                     [bboxcenter[0] * xfactor, bboxcenter[1] * yfactor, 
                      bboxcenter[2] * zfactor, 1.0])
    bobject.setMatrix(matrix)

Implementing glAlphaFunc in OpenGL ES2.0

2010-04-12T00:00:00+01:00

In OpenGL ES 2.0 glAlphaFunc isn’t available, you have to implement it in a fragment shader. There isn’t a lot of reference out there for this (not that I could find anyway) so I thought I’d write this up.

It’s actually quite simple to implement a quick alpha test. I needed to be able to cookie cut out sprites and fonts so I simply needed to reject fragments where the alpha value was zero. Here are the guts of a shader to do this:

#ifdef GL_ES 
precision highp float;
#endif 
uniform sampler2D u_tex0; 
uniform bool u_alphatestenable; 
varying vec2 v_texCoord;
varying vec4 v_color;
void main(){ 
    //calculate the fragment color based on the texture and the vertex colour
    vec4 basecolor = texture2D( u_tex0, v_texCoord ) * v_color;

    //if testing is enabled, check the alpha component and discard if zero      
    if(u_alphatestenable){ 
        if(basecolor.a == 0.0){ 
            //throw this fragment away
            discard;
        }
    }
    gl_FragColor = basecolor;
}

You need to set up a uniform variable u_alphatestenable which enables the alpha test. If you want to support different test types ( less than, greater than etc) then you will need two more uniform variables: one for the test type and one for the value to test against.

To enable the test you need to set uni_alphatest_enable to true:

int uni_alphatest_enable = glGetUniformLocation(mProgram, "u_alphatestenable"); 
bool alphateston = true; 
glUniform1i(uni_alphatest_enable, alphateston)

note that you shouldn’t call glGetUniformLocation every frame, it should be cached somewhere. It’s quite simple and while you may be thinking oh that is so slow it not that bad. It’s faster then the fixed function pipeline which is doing tests for alpha, lights, blend modes etc. If you get paranoid then you can create multiple shaders that support different subsets of features. All you need to be careful of is the cost of calling glSetProgram (to switch shaders) which can be expensive and cause bubbles in the vertex pipeline in the hardware.

Hope it helps.

dazza

Integrating Bullet Physics

2010-03-20T00:00:00+00:00

Bullet is an open source collision and physics simulation library written by Erwin Coumans who I believe worked or maybe still does work at SCEA (Sony Computer Entertainment America). Bullet provides a C++ API to a very robust Collision, Rigid Body and Soft body simulation system. At its core Bullet is comparable to commercial systems such as Havok but lacks the art pipeline integration, extensive documentation and support contracts that its commercial rivals offer. For an indie developer it’s an amazing leg up though. Collision is hard to do well. Physics is also hard to do well. Bullet does both very well.

Gradual Integration

Bullet has a layered compartmentalised design. So my plan was to first integrate collision and later Rigid body simulation. The documentation that Bullet offers is a little sparse but the source package includes a lot of demo code ranging from helloworld to a forkliftdemo to the classic how many boxes can I stack before the physics explodes. While it’s intimidating at first, collision and physics are so core to your game engine it’s important to spend the time integrating exactly what you need, where you need it.

Blender has Bullet integrated as well. Unfortunately not everything is exposed to the python export script. Rigid Body parameters to do with damping and most Soft body parameters are not available. The next version of Blender, which is a complete rewrite, should fix this. Unfortunately your export script will have to be rewritten as well.

Building the Bullet Codebase

The first thing I would recommend doing is building the bullet source package. Just follow the instructions in INSTALL. It’s worth installing CMake (as they suggest) as without it not ALL the demos will compile.

After compilation an executable called AllBulletDemos will be in in BulletSrc/Demos/AllBulletDemos unfortunately it doesn’t have all the demos in it but it does have a lot of them. Playing around with these demos will give you an idea of what Bullet is capable of and more importantly where to look for sample code. Bullet’s lacks hand holding documentation and you are expected to dig around in demos and find code that is useful to you.

Getting Data From Blender Into Collision

It’s well worth thinking about this quite carefully. Bullet recommends that you merge all static meshes into one big mesh. It also recommends that you don’t submit

large triangles (>10 units)
very small triangles
very long thin triangles.

In general the same routines you used in you blender geometry exporter will suffice for exporting the collision mesh. You should consider optimising the collision mesh to reduce the amount of data while remembering not to include degenerate (or near degenerate) triangles or overly large triangles.

In blender objects marked as Actors with Static selected are static collision meshes. To test for this is your python exporter:

if obj.rbFlags & Object.RBFlags["ACTOR"] and not obj.rbFlags & Object.RBFlags["RIGIDBODY"]: 
    #export the static mesh data

Bullet uses an OpenGL-ish method for submitting a large number of triangles for a mesh. This allows you to create an Index Vertex Array and from that build a Triangle Mesh Shape. One niggle is that the iPad uses 8 or 16 bit index values (GL_UNSIGNED_BYTE, GL_UNSIGNED_SHORT) whereas Bullet uses int (32 bit) indexes. This means you can’t submit your render index list to the collision system. In the long run you wouldn’t do this anyway but it would be nice to be able to use different index sizes.

mIndexVertexArray = new btTriangleIndexVertexArray(mNumIndices/3, 
    mIndices, sizeof(int)*3, 
    mNumVertex, (btScalar*),mVertex[0].v.x(), sizeof(ColVertex)); 
btVector3 aabbMin(-10000,-10000,-10000),aabbMax(10000,10000,10000); 
bool useQuantizedAabbCompression=true; 
mTriMeshShape = new btBvhTriangleMeshShape(mIndexVertexArray, 
    useQuantizedAabbCompression, 
    aabbMin, aabbMax);

Adding Meshes To Your Collision World

//Generic way to build and configure a Bullet Collision World with Dynamics 
mColConfiguration = new btDefaultCollisionConfiguration();
mColDispatcher = new btCollisionDispatcher(mColConfiguration);
mColOverlappingPairCache = new btDbvtBroadphase();
mColSolver = new btSequentialImpulseConstraintSolver;
mColDynamicsWorld = new btDiscreteDynamicsWorld(mColDispatcher, 
    mColOverlappingPairCache,mColSolver,mColConfiguration);
mColDynamicsWorld->setGravity(btVector3(0,-10,0));

Once you have created a world you need to populate it with you meshes. Bullet advises instancing meshes if possible and merging non instanced meshes into super meshes. When you instance a mesh you have an opportunity to use btScaledBvhTriangleMeshShape to scale the instance. You can’t scale btBvhTriangleMeshShape.

btVector3 localInertia(0,0,0);
btTransform bttransform = MatrixToBTTransform(transform);
btDefaultMotionState* motionState = new btDefaultMotionState(bttransform);
btRigidBody::btRigidBodyConstructionInfo rbInfo(mass, motionState,
    trimesh, localInertia);
btRigidBody *body = new btRigidBody(rbInfo);
mColDynamicsWorld->addRigidBody(body);

Since we are adding static meshes there is no local inertia and mass is set to zero (massless entities in Bullet are considered to have an infinite mass and are immovable). We construct a rigid body from the trimesh and the add the rigid body to the world.

Casting A Ray From The Camera

CML has a handy function called make_pick_ray which helps in the process of making a camera space ray which can be projected into world space using the camera’s transform.

void make_pick_ray(scalar_type pick_x, scalar_type pick_y,
 const matrix4x4_type &view, 
 const matrix4x4_type &projection,
 const matrix4x4_type &viewport,
 vector_type &origin, vector_type &direction, bool normalize = true );

[IN] pick_x and pick_y is the x position of the touch on the iPad
[IN] view is the view transform. I store the rotation of the scene in this matrix depending on the orientation of the iPad
[IN] projection is the projection matrix you have used to project from world to screen space. I make this using cml::matrix_perspective_xfov_RH
[IN] viewport is the matrix that describes how to move from normalised or device independant screen space to actual screen space. I make it using cml::matrix_viewport
[OUT] origin is where the ray is starts in camera space
[OUT] direction is the direction of the ray, it has unit length if you set normalized to true else it’s the length of the ray to the far clip plain set in your projection matrix. Once again it’s in camera space. In order to get the ray into worldspace you need to multiply both origin and direction by your camera matrix (note not the inverse one you use for projection by the matrix that describes the position of the camera in world space).

Cast The Ray Into the Collision World

btCollisionWorld::ClosestRayResultCallback rayCallback( campos, rayto); 
mWorld->GetColWorld()->rayTest( raystart, rayto, rayCallback );
if( rayCallback.hasHit() ) { 
    btVector3 colpoint = rayCallback.m_hitPointWorld); 
    btVector3 colNormal = rayCallback.m_hitNormalWorld; }

This really only covered the most basic form of hit testing. Adding capsules, spheres, cylinders, cubes, convex hulls and compound objects are all supported. I suggest you look at the demos :)

Summary

I’ve only really covered the basics here. Bullet is a massive piece of code and if you haven’t played with a physics system before I really think you will struggle. The forum is a help but tbh both collision and physics are hard so you shouldn’t expect it to be a walk in the park. If your time isn’t cheap it may well be worth paying for a solution.

Starting Out On The iPad

2010-03-09T00:00:00+00:00

Why the iPad?

When the iPad was announced I decided it was time to take the plunge and develop a game for it. Although the iPhone is a powerful piece of hardware I just find the screen too limiting. Every game leaves you wishing you had transparent fingers. The iPad seems luxurious with it’s 1024x768 A4 sized screen!

Having written many game engines over the years, both commercially and for fun I’ve run out of love for writing game engines. They are large and difficult to debug and I wanted to spend my time writing a game.

Starting Out

I looked for engines that I could buy in and I briefly evaluated SIO2, Oolong and Unity by going through the specs and reading this blog. I don’t really have any great technical insights beyond those in the blog and I plumped initially for SIO2.

Initially I was quite happy with SIO2. The tutorial videos are great and very instructive on both the use of Blender and the SIO2 codebase. After about two weeks I grew weary of the C code that makes up SIO2. It’s hard to move from C++ to C and the code felt very messy (it’s good C code but that doesn’t make it C++).

Facing the unhappy truth that I will needed to assemble my own engine I started to draw up a plan. The good news was that a large part of the work was assembly and plumbing. My intentions were to use Blender as a level editor, Bullet as a collision/physics system,OpenAL as an audio system and CML for maths. That’s a large amount of work imported from credible stable sources.

Week One Rolling My Own

The first thing I needed to write was an exporter for Blender and get the data into XCode and Rendered in OpenGL ES. Blender has a Python API and it covers most of what I needed.

scene = Blender.Scene.GetCurrent() 
objects = scene.objects 
for obj in objects: 
    if obj.getType() == "Mesh": #do some exporting of the mesh

For each mesh it’s just a matter of matching faces to materials and generating index, vertex, uv and color arrays for submission to OpenGL.

All in all it took about 4 days to get it all working.

The Red/Green/Blue axis you can see in the top right are tests to see that rotated instances are exported, imported and submitted to OpenGL correctly.

For four days work I was pretty impressed. It was all going really well :) So next thing to get working is transparency. Semi transparent objects need to be rendered AFTER the solid geometry and from back to front in camera space.

In practice this means that as the geometry for the scene is submitted to be be rendered any material with a transparent texture or alpha value is added to a list. After all the solid geometry has been rendered these ‘deferred’ strips are projected into camera space (their position relative to the camera is worked out), they are sorted from back to front and rendered in that order.

Each of the faces in each box is treated as a separate object to be sorted. Rendering from back to font makes sure that the correct geometry appears through the transparent material.

Next I moved on to collision and pyhsics…. Bullet to the rescue.

Replaying an Apache Logfile with Python

2008-10-01T00:00:00+01:00

NOTE: This is an old article I wrote in October 2008, it’s still relevant today. It was originally posted on luckydonkey.com which I am in the process of retiring.

In order to test for a memory leak in New Metal Army I needed to replay my apache log files against my test server. Using Twill this was a doddle.

The only slightly icky thing about the script is the regular expression to parse a line from the apache log file (in Combined Log Format). I got this from RegExp Buddy (pretty much the only reason I run Windows nowadays) but I am sure you can get similar expressions from other regular expression libraries.

Anyway, I’m just chucking this out there incase someone else finds it useful.

#!/usr/bin/env python
#Script to replay a file called apache.log against a server.

import re
import twill

test_server="my.test.server.com"

reobj = re.compile(r'^(?P<client>\S+)\s+(?P<auth>\S+\s+\S+)\s+\[(?P<datetime>[^]]+)\]\s+"(?:GET|POST|HEAD) (?P<file>[^ ?"]+)\??(?P<parameters>[^ ?"]+)? HTTP/[0-9.]+"\s+(?P<status>[0-9]+)\s+(?P<size>[-0-9]+)\s+"(?P<referrer>[^"]*)"\s+"(?P<useragent>[^"]*)"$', re.MULTILINE)
browser = twill.get_browser()

def filter_url(url):
    return False


for line in open("apache.log"):
    match = reobj.search(line)
    if match is None:
        continue

    f = match.group("file")
    p = match.group("parameters")
    d = match.group("datetime")
    path = "?".join([f, p]) if p else f

    url = test_server+path

    if(filter_url(url)):
        continue

    try:
        print d, url
        browser.go(url)
    except ValueError:
        #this comes from twill parsing the HTML and it being malformed.
        #I don't really care about that, as long as I get the page.
        pass

Hope that helps someone.

Python Style Plugins Made Easy

2008-01-28T00:00:00+00:00

NOTE: This is an old article I wrote in January 2008, it’s still relevant today. It was originally posted on luckydonkey.com which I am in the process of retiring.

Sometimes you need to write code that loads python at runtime. Plugin architectures are a good example of this. Plugins allow extensibility but more importantly (for me at least) they enforce a strict API. Anyway, I’ve written this code a few times so I thought I’d modularize it.

The specific bit of code I am going to post is the python code to look for and load a series of python plugins. Plugins (in this case) are just classes that are subclasses of the Plugin base class. This plugin base class dictates the API that all plugins must implement. Here is an example plugin abstract base class

class Plugin(object):
    def setup(self):
        """called before the plugin is asked to do anything"""
        raise NotImplementedError

    def teardown(self):
        """called to allow the plugin to free anything"""
        raise NotImplementedError

    def domagic(self):
        """do whatever it is the plugin does"""
        raise NotImplementedError

Here the is code to look for and return a list of classes that can be instantiated:

def find_subclasses(path, cls):
    """
    Find all subclass of cls in py files located below path
    (does look in sub directories)

    @param path: the path to the top level folder to walk
    @type path: str
    @param cls: the base class that all subclasses should inherit from
    @type cls: class
    @rtype: list
    @return: a list if classes that are subclasses of cls
    """

    subclasses=[]

    def look_for_subclass(modulename):
        log.debug("searching %s" % (modulename))
        module=__import__(modulename)

        #walk the dictionaries to get to the last one
        d=module.__dict__
        for m in modulename.split('.')[1:]:
            d=d[m].__dict__

        #look through this dictionary for things
        #that are subclass of Job
        #but are not Job itself
        for key, entry in d.items():
            if key == cls.__name__:
                continue

            try:
                if issubclass(entry, cls):
                    log.debug("Found subclass: "+key)
                    subclasses.append(entry)
            except TypeError:
                #this happens when a non-type is passed in to issubclass. We
                #don't care as it can't be a subclass of Job if it isn't a
                #type
                continue


    for root, dirs, files in os.walk(path):
        for name in files:
            if name.endswith(".py") and not name.startswith("__"):
                path = os.path.join(root, name)
                modulename = path.rsplit('.', 1)[0].replace('/', '.')
                look_for_subclass(modulename)

    return subclasses

and here is how you would call it:

classes = find_subclasses("./pluginsfolder/", Plugin)
#lets create an instance of the first class
inst = classes[0]()

So there you go, create folder of python files with classes in them that subclass the Plugin class and you are away.

The original article had a number of insightful comments, I’ve included all their corrections but unfortunately not the original comments. Thanks to all those who contributed to make this a better article.

PostgreSQL, SQLAlchemy, Dropping All Tables and Sequences

2007-11-23T00:00:00+00:00

NOTE: This is an old article I wrote in January 2007, it’s still relevant today. It was originally posted on luckydonkey.com which I am in the process of retiring.

There were a number of insightful comments in the original article that I have not copied across. They ended with The Ultimate DropEverything script from Mike Bayer that is now in the SqlAlchemy wiki

I’ve been working in deployment scripts for my current project and sometimes I need to drop all the tables and sequences from the database so I can create everything from scratch. I had been doing a

DROP SCHEMA public CASCADE;
CREATE SCHEMA public AUTHORIZATION bob;
GRANT ALL ON SCHEMA public TO bob;

This was great but a little destructive and over the top. It removes stored procedures and triggers as well which isn’t what I want. So I looked at the SQLAlchemy docs and there is a metadata command drop_all()

def drop_tables():
    metadata.drop_tables()

Unfortunately this doesn’t CASCADE so this isn’t going to work in cases where tables have foreign keys (which is most of the time). It also leaves sequences in place.

So I looked at dropping each table in turn with a cascade. So how do you get a list of tables and sequences in the database? I could hard code the names of the tables and sequences but that seemed a little poor.

It turns out that PostgreSQL has a set of views that expose the inner workings of your database and a few queries can give you all the information you need.

def get_table_list_from_db():
    """
    return a list of table names from the current
    databases public schema
    """
    sql="select table_name from information_schema.tables"\
        "where table_schema='public'"
    execute = metadata.execute
    return [name for (name, ) in execute(text(sql))]

def get_seq_list_from_db():
    """return a list of the sequence names from the current
       databases public schema
    """
    sql="select sequence_name from information_schema.sequences"\
        "where sequence_schema='public'"
    execute = metadata.execute
    return [name for (name, ) in execute(text(sql))]

def drop_all_tables_and_sequences():
    execute = metadata.execute
    for table in get_table_list_from_db():
        try:
            execute(text("DROP TABLE %s CASCADE" % table))
        except SQLError, e:
            print e

    for seq in get_seq_list_from_db():
        try:
            execute(text("DROP SEQUENCE %s CASCADE" % table))
        except SQLError, e:
            print e

The information_schema is full of interesting information. It’s actually a ‘view’ of lower level database tables. You can find all sorts of performance and configuration information in there. Quite handy Here are some docs.

Good luck.

FreeBSD Centralised Logging

2007-01-28T00:00:00+00:00

NOTE: This is an old article I wrote in January 2007, it’s still relevant today. It was originally posted on luckydonkey.com which I am in the process of retiring.

I wanted to log everything from my servers and router to a singer server to make it easier to keep up. Here are the steps I went through.

Edit syslog.conf

I’m using bonobo.local as my syslog server. Start by editting /etc/syslog.conf. At the first line that isn’t a comment (first line that doesn’t start with a #) insert + so for me I added +bonobo.local

#$FreeBSD: src/etc/syslog.conf
+bonobo.local
*.err;kern.debug;auth.notice;mail.crit          /dev/console
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err   /var/log/messages
security.*                                      /var/log/security
auth.info;authpriv.info                         /var/log/auth.log
mail.info                                       /var/log/maillog
... <snip> ...
!ppp
*.*                                             /var/log/ppp.log

then at the bottom of the file add the name of the client machine, for me this was hypnotoad.local:

!*
+hypnotoad.local
*.*                                             /var/log/hypnotoad.log

there are some funny characters in here that complicate matters.

!* resets the target. A few lines up the default syslog.conf has the line !ppp this tells the syslog daemon that all the following rules only apply to ppp messages. Putting !* tells the syslog daemon that all the following rules match all messages.

+hypnotoad.local tells the syslog daemon that the following rules relate to the machine mentioned.

*.* <tab> <filename> tells the syslog daemon to log anything from the afore mentioned machine to the logfile ‘filename’

For security reasons the syslog daemon won’t start appending (logging) to a file that DOESN’T exist when it starts so you must create the files before restarting the syslog daemon.

1	`sudo touch /etc/log/hypnotoad.log`

Edit /etc/rc.conf

edit /etc/rc.conf and append the line

syslogd_flags=""

This looks odd. Why deliberately pass in empty flags? Because /etc/defaults/rc.conf has

syslogd_flags="-s"

The -s option stops syslog from listening for external logging messages. This caught me out first time I tried to do this

Now restart syslog

1	`sudo /etc/rc.d/syslogd restart`

you should see a few lines about syslog stopping and starting again.

Setting up the clients

On each client make a backup of the original /etc/syslog.conf file and edit the original to include a single line

#$FreeBSD: src/etc/syslog.conf
*.*     @bonobo.local

replacing bonobo.local with the name / ip address of your logging server. restart your syslog server:

1	`sudo /etc/rc.d/syslogd restart`

try out the logging with

1	`logger "hello from my syslog client"`

the message “hello from my syslog client” should appear in the syslog servers log file for the machine you are changing.

Subversion and Apple Filemerge Integrtaion

2007-01-20T00:00:00+00:00

NOTE: This is an old article I wrote in January 2007, it’s still relevant today. It was originally posted on luckydonkey.com which I am in the process of retiring.

If you want visual diff’s while developing on OSX using subversion then this page has a good description of how to set it up and some scripts to wrap Apple’s FileMerge application to give to nice visual merges/diffs.

The author doesn’t mention it but rather then pass the path to the scripts to svn via the —diff-cmd option to can edit ~/.subversion/config and enter the full path to the scripts in the diff-cmd variable. Note it needs to be the full path as shell variables wont be expanded (so ~/bin/fmdiff will fail with error 255)

PS: I have nothing against the text based diffing tools so please don’t flame me. After all I’m a Mac user so what do I know of such magic.

FreeBSD: Slurm… the IPTraf alike for FreeBSD

2006-10-28T00:00:00+01:00

NOTE: This is an old article I wrote in December 2006, it’s still relevant today. It was originally posted on luckydonkey.com which I am in the process of retiring.

For many moons I have wanted an application like IPTraf for FreeBSD. Well through lack of trying and a certain amount of stupidity I couldn’t find one. All good things come to those who wait… to stumble across things late at night. I was watching an rsync backup happen across the network while reading FreeBSD Forums and I stumbled across Slurm. It’s not quite the same as IPTraf but its close enough… in fact in some ways it’s easier to use.

It’s missing the per port break down and the connection list that IPTraf has. But to be honest all I really want is ascii graphs . It is easier to use in my opinion because it just does what I want.. draws a cool graph and lets me know something is happening when I worried that I fucked something up again .

And you though this was going to be a late night rant about a fictional soft drink