NDA’d in 3… 2… 1…

Regular readers, subscribers (or stalkers!) may remember that I moved on from homebrew PSP development when I gained access to the official docs and development hardware through my work. Because my employer had signed a non-disclosure agreement for Sony, I couldn’t keep homebrewing and still guarantee that I wasn’t inadvertently leaking Sony’s uber-secret technical details to the general public.

I then moved on to GameCube development, as in work we weren’t doing anything for Nintendo hardware and I didn’t have to worry about those kinds of conflicts.

Until now that is, as we’ve just become registered WiiWare developers.

While we’ve not started development on any WiiWare titles yet, it’s probably just a matter of time. I’m really excited about the prospect of doing some cool stuff on the Wii, but I can’t help but feel a bit bummed out that yet another homebrew door has been closed.

It’s easy to feel angry at my employer about this, but really it’s not their fault. Sony and Nintendo’s attitude to developers is to blame here. But that’s a rant for another day.

If any readers would like to continue work on any of my projects, please let me know.

I’ll see if I can fire out a quick release of something before it’s too late, but no promises okay?

So long, PSP Quake, and thanks for all the fish

One of the problems with open source software is that it fundamentally conflicts with the requirements of working as a licensed developer. Up until now this hasn’t been an issue, but my employer is in the process of obtaining development kits and tools from Sony.

After I receive confidential information from Sony, I am contractually bound to keep it confidential. Unfortunately for Quake, this means that for the legal safety of myself, my employer and anyone who may use or view the PSP Quake source code, I cannot contribute any more to the project.

I can’t say I never saw it coming, but I was hoping our NDA wasn’t so strict. I was really hoping for a clause which would allow me to work on Quake without problems, as long as I didn’t use any of Sony’s tools. Unfortunately this is not the case.

What’s next for PSP Quake?

The project will continue, of course! I’m not the only PSP Quake developer, and I am sure that Chris Swindle will continue to do a fine job on the project, just as he always has. I wish him, and any future contributors to the project, the very best of luck.

What’s next for me?

In the short term, I have some rapidly approaching exams to study for. After that I’ll be splitting my time between work and whatever takes my fancy.

I will of course keep on homebrewing, but instead of the PSP I would be looking at platforms which we’ve not signed any agreements regarding. This could mean Nintendo DS, GameCube (GameCube Quake anyone?), Wii or maybe even XNA.

I would really love to port Cave Story to the GameCube or any other platform whatsoever, but my attempts to get in touch with the author, Pixel, have not been successful. Unfortunately I can’t speak Japanese and he can’t speak English! I tried emailing a translator he knows, but didn’t get a reply. Such is the internet.

If you can speak Japanese and would like to help me out, or can help in another way, please get in touch.

Teamwork, clipping and optimisation

Apologies for the lack of updates, I’ve been rather busy with study.

I got back in touch with Chris Swindle, and we’re working together on the game again. We’ve made some really amazing progress over the last few days (or in my case a few hours late at night).

So what’s new?

Clipping is in. Now there are no more missing polygons when the player wanders around.

This was a bit of a pain to do, but thankfully it’s done now. The only real difficulty was that while the majority of the world polygons are in world coordinates, brush models (like ammo boxes and elevators) are in object coordinates and require a different clipping frustum.

So at the start of the frame, I build the clipping planes from the view and projection matrices. When I need to draw a brush model, I combine the model matrix with the cached view and projection matrix, and calculate the new clipping frustum. After rendering the brush model, I restore the old clipping frustum.

Clipping is one of those things that’s hard to debug. You either get a properly clipped scene, or a blank screen because your clipping planes are broken. Knowing how to do clipping like this is guaranteed to come in handy for work in the future, so I’m really glad it’s in and working.

Additionally, all kinds of optimisation has been done! Thanks to Chris’ knowledge of gprof, we can now profile our builds and find the bottlenecks. As a result we’re now at about 65 frames per second, depending on the demo being played back.

There’s a load of clipping math which I’d like to offload onto the VFPU, but at the moment I’d rather fix bugs and implement required features than optimise.

We’ve still got a few things like light maps to implement before the game looks like Quake, so I added them to the SourceForge tracker to stop me forgetting!

Hopefully light maps won’t drop the frame rate too much, but we’ll cross that bridge when we come to it.

Screenshots blitz

Difficulty selection

Yes, I’m a wimp that plays on Easy!

Episode selection

Presumably most people will play the shareware version.

Sneaking my way round the first level

Cue elevator music.

Hello nice doggy… “wuaaaargh!”

Secrets? Real men don’t stop to power up.

Or like me, they’re too busy running screaming for the exit.

The second level

Hmm, it’s probably time to move clipping to the top of the “to do” list.

A brief encounter with an Ogre ends badly.

Performance

I ran a little “timedemo” test earlier today to see how I’m doing in terms of frame rate. Demo 1 (The Necropolis) runs at 38.3 frames per second on average – a good bit better than the software version, which ran at about 24 frames per second, at only 320×200.

There’s still plenty of optimisation to go, so I’m hopeful that I can get it to run at 60 frames per second. At the very worst (i.e., if light maps and clipping slow the rendering down a lot) it’ll run at 30, but I’ll try my best to improve on that.

Most polygons don’t need clipped. Hopefully if I have a quick enough “does this polygon need clipped?” check, there’ll only be a noticeable increase in processing time on polygons which require clipping. If that’s only for a couple of polygons per frame, it should be invisible.

Light maps are a bit of an unknown. Essentially I’m going to have to clip and render the world polygons again, with modulative blending this time.

The additional texture reads are probably going to require some texture optimisation to keep texture bandwidth under control, and the extra frame buffer load (combined with the shortage of VRAM) may push me to move the frame buffers to 16 bit.

What’s next?

I’ve got a fairly lengthy “to do” list on the go. A work colleague from a previous job had a way of organising these things, which I can’t seem to forget about. As a result, my “to do” lists always seem to take on this form!

Must do before next release

These are things which need to be done for the game to either look correct or not crash. It’s fair to expect these in the next release.

1. Clipping.
2. Draw underwater polygons. Currently only polygons above the water line are drawn, which will obviously need changed.
3. Light maps.
4. Deallocate texture memory after levels.

   GLQuake doesn’t seem to free textures, so I’ll need to hook this in somewhere or the game will probably run out of memory and crash after a few levels.

5. Networking.
6. On screen keyboard.

The last two features on this list were present in the previous release of Quake, but had to be removed to both restore the cross-platform code’s cross-platform status, and make the GLQuake porting effort easier.

I intended to branch PSP Quake and do an XGP port, requiring the cross-platform code to be free from PSP-specifics, but Game Park subsequently went out of business before making the console available. Their website now sells shoes!

But anyway, that’s why I’ll need to reimplement those tasks.

Should do before next release

This list is for things which would increase performance, or improve the appearance of the game. You can probably expect some of these to be done in the next release.

1. Increase console text width
2. Make particles look better
3. Optimise particles
4. Optimise vertex formats
5. Light models
6. Swizzle textures
7. Optimise vram usage

Potential improvements

These are things which probably won’t make it into the next release, but may be added if it’s quick to do, doesn’t break anything, and provides a reasonable benefit.

1. Texture atlases for small UI textures.
2. VRAM cache.
3. Fog under water.
4. Preset list of game servers.
5. Update Win32 stubs for GU calls.

I don’t personally play Quake online, but I know people would like the process of joining online games to be made easier for PSP users.

Please let me know via the comments if you would like anything added to my task list. All suggestions will be considered.

Progress

This entry and the 3 which follow it were originally one mammoth post, but I trimmed it down. Thanks go out to WordPress for deleting 2/3 of the text when I tried to add a screenshot.

Even though I’ve got an essay due in a week, 9 upcoming exams, and a job to keep on top of, I found some time late last night to work on the game some more.

Getting to know the PSP graphics system outside of work hours is arguably beneficial to my employer, but I wonder how my boss would see it!

Most of the time was spent fixing up the Makefile, primarily ensuring that only files which are out of date get built. The areas of the Makefile pertaining to C and C++ code were fine, but because the Makefile implicitly used some PSPSDK rules which didn’t have dependencies, some files always got built, even if nothing had changed.

Specifically these were kxploit-related files, which only need built when the game icon or XMB background are updated.

As a result of the dependency tweaking, the game is now only copied to the memory stick when it’s changed, and only the changed files are copied.

I then went on to fix a nasty model shearing bug caused by some “optimisation” in the SDK’s sceGumScale function. Oh boy, that one took some time to track down! That will teach me to unconditionally trust the SDK. Anyway, now the gun model is displayed properly and models aren’t off centre or squished.

Additionally, I did some work on the viewport code, meaning that the user can resize the display window from the options menu, and the renderer handles that properly. Up until now, the aspect ratio was off, and the screen would start to warp if the user changed the size.

There’s no performance or appearance benefit from reducing the screen size smaller than what is needed to reveal the whole status bar, but the user can do it anyway, just like they could in GLQuake.

I also tweaked the Visual Studio project file, removing an unused configuration and putting both the PSP version and the Win32 stub version in the same solution.

All in all, a pretty good night of coding. The game is starting to look and play like Quake, instead of being a load of glitches strung together with Quake sound effects!

No more Quake today though, I have to work on my essay.

Texturing progress

After flat-shaded polygons were up and running, I thought I should work on texturing. Adding textures isn’t as simple as one may first think.

Memory allocation

While previously the only things stored in VRAM were the frame buffers and depth buffer, textures will require dynamic allocation of VRAM. This prompted me to write a simple VRAM allocator.

The allocator works by splitting VRAM up into fixed sized chunks (currently 32×32 bytes). When the renderer needs VRAM, it calls the allocator’s allocate function, which searches through its list to find a large enough sequential block of chunks. It then tags these as allocated.

Obviously this has a couple of potential problems:

• Small textures waste memory.

  Because each block is 32×32 bytes, textures which are smaller than this waste space. On the PSP this isn’t a big deal, because the hardware can only handle textures where the pitch of the buffer is 16 bytes or more. This generally means that small textures have to be padded out anyway.

  Additionally, Quake textures tend to be roughly this size. Some smaller textures (e.g., 24×24) need to be scaled up to 32×32 anyway to have a power of two size.

• Large textures take up several blocks, leading to free space fragmentation.

  This could be a bit of a bummer. I would really like to avoid fragmenting VRAM, especially because we don’t want to have to waste time defragmenting VRAM when we could be rendering.

I’m sure my allocator could be smartened up and optimised, but it works for now, and that’s what counts. After all, there’s no point in optimising the texture allocator right now, before I know what kind of allocation patterns I’m going to end up with.

Limited memory

At the moment, Quake needs about 3MB of VRAM for the frame buffers and textures. The PSP only has 2MB. As a workaround, when I run out of VRAM for textures, I allocate from system memory instead. Performance is still fine for now, but I will probably need to add a VRAM cache system (probably LRU, LFU or some kind of hybrid approach) later.

Currently the frame buffers take a little over 1MB (1088K), and the depth buffer takes 272K. This leaves only 688K for textures, which is a bit rubbish. I may end up moving from 32-bit frame buffers to 16-bit to save some memory and reduce VRAM cache misses.

I will probably end up converting textures from 8-bit to DXT1 if I need more VRAM and the compression artefacts aren’t too bad.

I will also want to swizzle textures, making them faster for the PSP to use.

But for now, performance is adequate. I’ve not done any benchmarks, but the frame rate is visibly smoother than the software rendering version, and there’s still plenty room for optimisation.

Oh, one more thing, no developer diary post is complete without a screenshot:

Things have moved on some more since I saved that picture, but I’ll leave that for next time.

Work on hardware rendering is underway!

It’s been a heck of a long time since Chris Swindle and I did a Quake release, but hopefully there are still some people out there enjoying it.

I managed to find a few hours this week to work on hardware rendering for Quake. It’s a long time overdue!

When deciding how to go about writing the hardware rendering layer, two potential approaches immediately sprang to mind:

1. Hack the software renderer, adding hardware replacements until it’s done (or fast enough).
2. Rip the OpenGL calls out from GLQuake and replace them with PSP GU calls.

In a perfect world, both paths would arrive at the same destination. Both approaches would work pretty well, but after some deliberation I decided on the second approach. GLQuake has covered much of the groundwork already, so I should probably capitalise on that.

A worthwhile diversion

The initial worry was that I would hack away at the GLQuake code, commenting out OpenGL calls and/or replacing functions with stub (no-op) versions, compile my code, run it, then immediately hit a null pointer exception and crash. And of course, that’s almost exactly what happened. The game started, got part of the way into initialisation, and the PSP powered off. (This is what happens if you don’t handle a hardware exception.)

Now, debugging on the PSP is a bastard. Especially if, like me, you’re not a Linux type of guy and wouldn’t touch GDB with a 10 foot barge pole. Maybe official developers get a great debugger, I have no idea. I’m using the homebrew SDK here.

I then had an epiphany. Visual Studio’s debugger kicks ass. A running screaming arms-waving leg swinging punt right up the sphincter. Getting this crash to happen in Windows would make it a damn sight easier to fix, so I set about making it so.

Inspired by some “work” work I’ve been doing recently, I whipped up some minimal stub versions of the PSP SDK headers I was using, and created a Win32 build of the project. Obviously the functions, like most, do nothing visible to the calling code. They do however allow the calling code to compile, link and run.

With no actual graphical output, but with the Quake engine and most of my port code humming away in the background, I built and ran the Win32 version. And like all good bugs, this one appeared promptly, making the whole Win32 build effort worthwhile. Incidentally, it was one of the worst kind of bugs – the subtle typo that many a coder has spent dreary late nights hunting for. After deleting a single misplaced “!” character, the bug was no more.

So back to the task at hand

After the game successfully ran on Windows and PSP, with a whole lot of GLQuake running but without any PSP rendering going on, I went back to what I was supposed to be doing. I added some GU initialisation and buffer swapping code, just to get something up and running.

I slowly adding drawing code, until I had enough to draw flat shaded polygons with “random” (so that’s what a pointer looks like when reinterpret_casted to ABGR) colours. I also went about replacing the OpenGL matrix operations with GU and GUM equivalents. And it worked! What a stroke of luck, but damn was it pug fugly:

And that was the end of another development day. Thanks for reading, there’ll be more in my next post.