The GCNrd Story.

The most important piece of software I ever released was a Nintendo GameCube debugger over 20 years ago. It was not a financial success, receiving roughly $200 in donations. The initial version was released in full and for free on 2004-03-31, around 9 months after development began.

The Seed Planted

I was a bit of a boaster. After several years of creating cheat device codes^[1] that were a bit more creative than the standard flare of "infinite health" or "start on level X", I had given myself license to make bold claims without any real need to back them up. I had a knack for finding the developer's hidden debug menus and changing game physics in unintended ways, allowing players to explore the game world outside of the predefined boundaries. So, one day in 2003 when I claimed that I was going to do something to shake up the game hacking community by creating a GameCube debugger, a lot of people probably accepted it without thinking twice. On the other hand, I had no idea if I could actually pull it off.

A few similar tools had already existed for other platforms. The most popular at the time was probably Caetla. Connecting your PlayStation through an ISA "Comm board" and an original Action Replay / GameShark enabled full control over the PS1 software including a debugger, disassembler, and some homebrew development tools. It was an obvious inspiration for GCNrd.

Action Replay was released for GameCube on 2003-02-18. And this, in my opinion, is where it all began. Datel had already done all of the legwork to break into the system and execute arbitrary code. Not much is known about how it all happened, or even who was involved. The biggest problem for the game hacking community was that the codes were all encrypted, and only Datel (CodeJunkies) had the tools to create codes that would be accepted by the AR. Inputting random characters never worked. The AR would always be able to determine it was not a legitimate code, or it would notice that the code was valid but was for the wrong game.

For the first few months, nothing really interesting happened. CodeJunkies continued to very slowly release new codes, usually only one batch for whatever the newest game was at the time. Older games were mostly ignored after the first batch of codes was released for them. If you wanted a specific kind of code for your favorite game, you were out of luck and at the mercy of CodeJunkies to do what they wanted to do.

On 2003-06-17, everything changed. A program called PSUL^[2] was released by CrazyNation (legendary N64 sceners) that allowed anyone to execute arbitrary code through a broadband adapter (BBA) and the Phantasy Star Online Episode 1 & 2 game. In mere hours a number of game disc images were also released in the warez scene. One of them was a version of the Action Replay.

I saw this as an opportunity. After a few days of reverse engineering the PowerPC assembly from the AR dump, I released a tool that was capable of decrypting existing AR codes and encrypting brand new "homebrew" codes without Datel's permission. GCNcrypt released on 2003-06-22, just 5 days after the PSUL release. Not bad but could have happened sooner if not for all of the weird stuff Datel did with the AR codes. I had to interpret the meaning of the "identifier code" which is one of the ways the AR rejected random inputs. Real life was also slowing me down, I had work and nothing to go on but intrinsic motivation.

According to my notes, it wasn't until two weeks after the release of GCNcrypt that I decided to actually make the debugger a reality. Two months into the project, I started working on the network driver that would become the foundation for the debugger. Most of it was reverse engineered from the official Nintendo SDK and compared to datasheets available for a very similar (though not identical) Macronix ethernet controller. Within 5 days, I had enough working that I could send and receive UDP packets to a PC connected on the same network. This was 2003-09-01, and though I worked on the network driver mostly on my own, I was also moonlighting as a scener, releasing GameCube ISOs and writing tools for the group to use.

One of the sceners I worked closely with at the time was Aftermath. He and I came up with an idea to run the ISOs on GameCube hardware by streaming parts of it over the BBA. And after the network driver was usable enough, that's exactly what we did. We picked the easiest possible game to start with, Animal Crossing. Because it was a small game, and it was known at the time that you could remove the disc after the title screen and the game would just keep running forever. Everything it needed was loaded at the start, and then it would not need to read the disc ever again. It was the perfect test case. Animal Crossing Loader was released on 2003-10-10. It was the first publicly available method of loading ISO backups on the GameCube.

ACloader went through several iterations, starting with compatibility improvements, and each release fixed bugs or increased network performance. At its core, the idea was simple. After initializing the network driver and setting a fixed IP, it would send packets to a predetermined server IP to establish a session. The GameCube would request the initial boot blocks from the virtual disc, which the server would send as requested. It followed the basic boot process, reading the boot block and running the AppLoader, etc. After the main executable was loaded by the AppLoader, the memory would be scanned for the Read() function, which would be patched to read from the simulated DVD drive over the network.

The debugger worked on the same basic principle, so it's no wonder GCNrd shares a common ancestry with ACloader. They share about 80% the same code, which we called "libeur". The initial GCNrd release in 2004-03-31 removed all of the "DVDsim" capability, which I kept in private builds. GCNrd was most well-known as a debugger for commercial game discs, but it played a role behind the scenes that made it possible to release new codes as soon as a game hit store shelves, sometimes even earlier.

Sceners raced each other to get the newest games as quickly as possible, and sometimes that meant a scener embedded somewhere in the logistics for game distribution would end up taking a disc home prior to the official release date where it would be copied and uploaded to FTP sites across the internet. On the other end, I would download the ISO and run it locally with the debugger's DVD simulator. I would create codes and designate a new unique game ID for it. In some cases, Datel would adopt the same game ID for their official code list.

The Nitty Gritty Details

It wasn't all roses and rainbows. Though what I was able to achieve with GCNrd is hard to overstate, the saga was laced with drama and technical challenges. Aftermath ended up doing a large number of ACloader-spinoff projects on his own, infrequently contributing his improvements back to GCNrd. Some of them are lost forever. I had a contentious relationship with Costis, of PSOLoad and SDLoad fame. I made some very minor contributions to SDLoad, which made launching GCNrd about 100 times faster. Datel copied the idea for their own SD Media Launcher about 2 years later. LOL.

Development with the PSO loaders was very rough. It took about 2 minutes from power-on to first Arbitrary Code Execution. Several compatibility bugs that GCNrd had were due to state leftover from PSO. Typically, some hardware needed to be reset before starting the new game. Star Wars: Rogue Squadron II - Rogue Leader was the one that gave me the most trouble. I would spend several hours across multiple days trying to find the cause of the game crashing in the level intro. I think I eventually fixed it, but I no longer remember what the actual problem was. And even then, SDLoad didn't have the same crash bug. So, I may have not fixed it at all, instead recommending SDLoad as "the right way" to use GCNrd.

Programming in assembly and low-level C for the console was harsh. It was too easy to screw things up, especially because there is no cache coherency in the CPU. Doing any kind of self-modifying code was a unique challenge. There was no real debugging, apart from what GCNrd itself implemented! I could send formatted strings in UDP packets, which is how I did a lot of the initial development. And ultimately, I added an exception handler to catch crashes and report thread context. It was a slog, lots of time resetting the GC and waiting through the PSO loading screens. Debugging was just trial and error for months on end.

I had limited memory available, about 32 KiB which I reserved for the debugger and its network stack. I was able to offload some of the logic to the PC-side by implementing commands that could read and write arbitrary memory locations. This was probably the saving grace for the project. The memory-resident debugger could get away with just the bare minimum functionality for using a pre-initialized broadband adapter (the menu would initialize prior to loading and running the memory-resident half) and some basic functions to control the debugging circuits on the CPU. Most of the interesting parts of the debugger were built on top of these simple primitives as a "remote debugger". Searching for library SDK functions just needed to read memory, for instance. The comparison code existed on the PC-side. Patching was the same, the assembly routines were on the PC-side and the only thing needed on the GC-side was the capability of writing to memory.

In hindsight, it would have been much easier with a smaller memory footprint to create a serial interface instead of using the BBA. However, I had good reasons (at the time) for preferring the BBA:

1. I lacked any kind of hardware knowledge. Designing even a simple serial interface at that point was not something I could have done. In 2004, it was unclear if the GC even had a general-purpose serial port available to use (e.g. not the controller ports).
2. The BBA was already available, and it was much faster than what could be done with a bespoke serial port. The full 24 MiB of main memory could be downloaded in about 14.5 seconds, according to a pre-release note that I kept in my logs. A rate of about 1.7 MiB per second. This was about half of the max theoretical bandwidth of the EXI bus.

Source control was basically non-existent at the time. Mercurial and Git wouldn't have been released for another 2 years. Code was copied to other directories or zipped to make backups, whenever I felt like it. Which clearly wasn't enough. I no longer have the code for the initial released version, or the second release, "v1.10b"^[3]. I instead have a few variations of source code from intermediate points between (and after) both releases. One day I would like to go through these variations and try to understand what was changed between them.

I could release the code, but I don't even know if it compiles. There was a point in time when I thought I would clean it all up and make the memory-resident debugger relocatable and remove all of the duplicate code between the menu and the debugger. That subproject was incomplete, and I am sure it left the code in an unusable state. The code is also absolutely terrible, being a specimen from my earliest developer days. It would be a really bad release without some real curation effort going into it first.

Wrapping Up

So, why write about this now? Well, for starters, it all happened a very long time ago. Everything about GameCube is already known by now, but back then we were pioneering. I left the warez scene sometime in 2008 because I felt like I had done everything I set out to do. I made piracy possible on GameCube and had a pretty big impact on the DS as well. These days, game piracy is completely alien to me. I just felt a need to reminisce about a very interesting time in my life, at least the good parts!

It's very weird watching something like an MVG video where he describes your work but doesn't say anything about who you were or how you did it. Clearly, I had an impact on many people's lives, and they don't know anything about me. The ACloader was built mostly in anonymity (for obvious reasons)^[4], but I'm happy to claim responsibility for it. It was an awesome hack that let hundreds of thousands of people^[5] play games that they would otherwise not have access to. On the other hand, GCNrd was always associated with my internet handle (Parasyte), just stripped down to remove the piracy features.

Given the chance, would I do it again? In a heartbeat! It's as close as I have ever come to feeling like my existence was meaningful. That I wasn't just consuming resources, but also producing something of value. It's what I imagine the early phreakers and hackers must have felt in the 60's and 70's, before I was born. Doing something that was generally looked down upon, but in its own right created immense value by allowing people to connect in ways they couldn't before.

I am proud of GCNrd and ACloader. And even though I have a chip on my shoulder because I was never able to top those projects (despite untold amounts of effort) I still believe it was one of my crowning achievements. I was 20 years old, self-taught, working a terrible low-wage job, and I just personally enjoyed messing around in game internals. It was inevitable that I would create tools to help other people do the same thing.

All that said, GCNrd and ACloader was not my only project that gained wide attention. I hinted about DS earlier, but I also did a lot of stuff with N64, NES, SNES, PS1, all of the GameBoys, and dozens of emulators for several platforms. It's too much to list, but there are a few other good stories in there that have never been told. Maybe someday I'll feel nostalgic enough to write about them, too.

In summary, working on the project was miserable at times and it's now largely forgotten. It was all worth it, though.

---

1. "Cheat devices" were mostly unlicensed cartridges or discs for home video game machines that would allow players to make some real-time modifications to games. The most common effects were "cheats" like being able to take an infinite amount of damage or give yourself all of the unlockable upgrades in a game.
   
   
2. As far as I can make out, PSUL and PSOSocks were both released on the same day. PSOSocks contains a disc dumper and other tools. It kicked off the GameCube piracy scene. At least as far as copying went. PSUL was more homebrew-oriented, and it can be said to have kicked off the GameCube homebrew scene.
   
   
3. I DO have the complete source code for the very first ACloader, however. Which probably deserves a release just for the historical significance.
   
   
4. The initial ACloader clearly lists "AFTERMATH AND PARASYTE" in the scroller, and the PC-side server includes the same information. But I tended to distance myself from it in the open.
   
   
5. I really don't know how to estimate the actual number, but 6-digits is easily in the right ballpark.

Goto is an Abomination

In response to Goto is Not a Horror, I must implore you witness:

func main() {
	var i = 0

	loop:
	i += 1
	if i > 10 {
		goto done
	}

	fmt.Println("Hello, world", i)

	goto loop
	done:
}

You might say, "that isn't fair, this is just bad code using bad practices". And you would be half correct.

Goto can be used for evil. And it is. Too often, goto is abused. But here's the thing: Goto can only be abused. The example is fair criticism because there is no "good" use case for it.

That was Dijkstra's point.

The go to statement as it stands is just too primitive; it is too much an invitation to make a mess of one's program.

It's too unconstrained, or "unbridled" as Dijkstra put it. This is still the case with the goto keyword today. And it will remain the case 56 years from now.

"It's not a problem anymore, that goto is gone," says this unusual justification for an abomination, as if Go supposedly fixed it. You've clearly read the original 1968 letter, but seem to have missed its 1987 follow up, "On a Somewhat Disappointing Correspondence."

The real myth is that somehow jumping around arbitrarily just magically stopped being an issue. As the 1987 paper alludes to, we have better problems to solve than your decision to avoid constraint.

You can write code without using the goto keyword at all and it will make your code less enigmatic. Try it sometime. Stop defending the indefensible.

As for me, I thought that by now professional programmers would have learned something.

Improving build times for derive macros by 3x or more

I was dissatisfied with argument parsing crates. There didn't seem to be one that checked all the boxes I cared about:

• Simple. Don't overcomplicate it! All I want to do is provide a rudimentary human interface to my app. If I wanted interface complexity, I'd put a whole TUI library in it.
• Don't make me repeat myself, and don't make me write unnecessary boilerplate. The only thing a parser needs to do is compare given args to a known list and populate some state based on it. It shouldn't require adding a bunch of attributes to define how that state is populated, and it shouldn't require writing the application "help text" that describes the arguments separately from struct declaration.
• Correctness. If I give my app invalid UTF-8 and it panics or returns an error inappropriately, I'm going to be furious.
• Small! I once used clap as a young Rustacean and a third of my binary size was for parsing arguments! The app was fairly sophisticated, doing things like parsing ELF files. A whole third of the size of the binary was only used in the first few milliseconds of the app's entire runtime. (I may be exaggerating a bit, and I can't find any evidence to back the claim. The very first commit in the project in question doesn't contain any argument parsing library. No doubt due to the experience described.)
• Fast! Build times are terrible for all arg parsing #[derive] macros that I've seen. Even on my relatively modern/high-end desktop (with a Ryzen 9 5900X running Windows 11 and Ubuntu 22.04 in WSL2), the fastest derive crate (argh) takes 2.15 seconds for the initial build. For comparison, the fastest all-around crate is pico-args at 370 ms.

There's a list of popular argument parsing crates along with various benchmarks at https://github.com/rosetta-rs/argparse-rosetta-rs. These results sparked my curiosity. If we eliminate all crates that don't support derive and invalid UTF-8 from this list, we're left with just two options: bpaf_derive and clap_derive. Both of these fail the "small" and "fast" requirements, though bpaf_derive is the better of the two.

Stability and versioning: Lock yourself in at your own peril.

I usually write about Rust, but today I want to discuss something broader; flexibility in software, particularly in design and architecture. To begin, let's focus briefly on versioning.

Versioning

Prior to semantic versioning, version numbering conventions used in software releases were nearly as numerous as the releases themselves. Some had implied meaning for releases with odd vs even numbers, some would skip numbers arbitrarily, some would use letters and other non-numeric characters, and some would have unique counting strategies. Semantic versioning provided some much-needed consistency.

Consistency, however, is not the ultimate goal. You might imagine some issues with consistency itself; is it better to have some inconsistencies (mix the good with the bad), or just be consistently bad? One of the most important goals for a software maintainer is to provide repeatability; in other words, consistency within your build process and testing infrastructure. Same word, just scoped differently. Versioning, when used appropriately, can help increase the repeatability of your builds and tests.

So then, let's defined "appropriate versioning". If we follow SemVer closely, then it should not be possible to introduce breaking changes in our software that will negatively impact downstream dependents. If a mistake is made and a breaking change goes out in a version that is not a semantically breaking version, then we have to do the right thing and unpublish/yank the semantically incorrect version. That's all there really is to it. I'll leave you to read the SemVer spec for the precise rules to follow, but this is all that one needs to appropriately version their software.

Stability

This term, "stability", is ironically such a fragile concept. For something to be stable, it must be unchanging. For something to be unchangeable, it must be perfect. Very few things are universally perfect. I like to use science to illustrate this point: the sciences are ever-evolving; knowledge is not static but accumulates and is refined over time precisely because it didn't start out perfect. It is fitting that computer science is one of these fluidly changing organizations of knowledge.

When software engineers think of stability, they are usually looking at it in the context of any of the following layers of abstraction:

• API stability. This is what SemVer addresses directly.
• ABI stability. The idea that binaries compiled long ago (or by different toolchains) should work transparently with binaries compiled today.
• Resilience. Resistance to changing behavior or breaking compatibility, but also resistance to logic bugs and the like.

Resilience in service-oriented software is a very interesting topic of ongoing research, but we can frame resilience in the terms described above. Namely resistance to logic bugs, since changes in behavior or interface are better addressed through tools like SemVer.

So, what do we know about logic bugs in stable interfaces? The first thing that comes to my mind is in the C standard library. It's our old friend, the null-terminated string and its wild band of merry havoc-wreaking unsafe functions! There is good reason this has been dubbed the most expensive one-byte mistake. Now just because C has null-terminated strings and its standard library supports them doesn't mean you have to use these abominations even if you write code purely in C. But the sheer support for them in the wider ecosystem sure does make it harder to integrate various libraries when you are using any of the safe alternatives. Not to mention the fragmentation that each of those contributes to.

We are left with a stable interface whose intractable design makes it almost useless in practice. It may as well not exist at all! We'd be better off without null-terminated strings, de facto. But in some sense, we are stuck with them for all time, and there is nothing that I, you, or anyone else can do about it. This is clearly a highly important lesson of stabilization.

There is another term for stability, often used with critical connotation:

Ossification

No doubt, many will first encounter this term when trying to understand the QUIC protocol. It has been used in various circles for far longer, though.

• Here it is seen in an article written in 2007 on the state of the Internet at the time: "The Internet is Broken"
• And here it is used in reference to the C++ standard library: "The Day the Standard Library Died"
• And Fastly mentions it in regards to the Y2K bug (among other examples): "Minimizing ossification risk is everyone’s responsibility"
• Of course, it also makes several appearances in the Cloudflare Blog.

If you read between the lines a little, ossification sounds like an objectively bad thing. It is outside of your control, and you cannot fix it. What could be worse than that? In my opinion, being the decisionmaker in that case would be worse. I wouldn't want to be held accountable for saying "this is our call; it's set in stone and it can never be changed."

Stability comes packaged with this gremlin called ossification, and they are inseparable. What sounds like a good idea right now may turn out to be a bad idea later. It happened with null-terminated strings and nullable pointers, it happened with PHP and Python, and it will happen again. Maybe the next library or application you stabilize will turn out to be totally wrong/broken/inadequate.

You can't tell the future, but you will definitely be stuck with the past.

Outside of standard libraries that ship with programming languages and compilers, it is a pretty safe bet that being totally wrong/broken/inadequate is a reasonable situation to be in; you just patch your library or application when things need to be fixed and all is right, right? I think this actually depends on a few aspects:

1. The software maintainer needs to be on top of maintenance. If they fall behind on making patches or merging PRs, then you're left with the maintenance burden yourself, often on software that you are not an expert in. It just happens that you depend on its functionality and you get stuck with the bill when the maintainer moves on to other projects.
2. Other dependents need to maintain their software, too! That means all transient dependencies in the entire tree need the same treatment and amount of prioritization to receive patches in a timely manner.

To echo the sentiment of the Fastly article, as long as everyone keeps up their end of the bargain with maintenance, there isn't a whole lot that can go wrong. It's just an impractical task to herd all of those cats; that's the reality.

What, then, is there to do about standard libraries? Hopefully as little as possible! Considering the likelihood of getting a design or public interface wrong, and the resistance to change with strong stability guarantees, it is in everyone's best interest that standard libraries be as small and self-contained as reasonably possible. Even some primitive types that are useful in general may be better off in a library that can freely be updated (in breaking fasion) by end users. Yes, I'm thinking of strings in C, again; but I'm also thinking about mutex poisoning in Rust; I'm thinking about Python's multiple XML interfaces and its asyncio module that is arguably poorly designed compared to curio and trio; I'm thinking about Go officially supporting largely irrelevant and insecure ciphers like RC4, DES, and MD5 (honorable mention to both pseudorandom number packages, because that's never going to confuse anyone).

Maybe it's better to not include everything in the standard library. We just end up hearing recommendations over and over about how you shouldn't use this feature or that function because it's unsafe/insecure/deprecated/there's a better alternative. Starting with a very small surface area seems like the right way to go. This is not a proposal to start small and grow gradually, though! I don't think a standard library should grow much, if at all.

Think about it this way: Something like JSON might seem like an obvious choice for inclusion now, but in 40 years, is anyone honestly still going to use JSON? I mean, ok, fine. People are still using COBOL. But my point is that 80% of all developers are not COBOL devs. In four decades, sure there will be some JSON stragglers just like there are some COBOL stragglers today, but I can't believe that it will remain popular considering how bad it is for so many reasons. (Let's face it! The only reason protobuf or something else hasn't entirely supplanted JSON is because web browsers have ossified their serialization format.) JSON will very probably end up like null-terminate strings; the bane of some seasoned developer's existence. They both seem just so innocent, don't they?

Be very cautious about what you choose to ossify.

Resilience

I'm not much of a words-person, but when I research things that I write about, I usually go to the thesaurus often to find a good synonym for whatever concept I'm trying to convey. One of the top synonyms given for "flexibility" was "resilience"! And this was a fascinating discovery itself, worthy of a paragraph or two. Flexibility implies resilience because flexible things bend rather than break. When something is rigid and enough pressure is applied, it has to break at some point. And breaking is not very resilient, is it?

This definition of "bend rather than break" applies in a number of relevant ways for software design. A service provider strives for resilience to downtime and outages. A programming language designer and compiler implementer strives for resilience to breaking user code. Interface designers strive for resilience to footguns and obvious logic bugs. In each case, the person wants the software (or user) to bend rather than break.

I like it. In fact, I'm willing to boldly claim that resilience is objectively desired. If that is the case, then its synonym, flexibility, should also be just as desirable, objectively. Among the definitions of this particular word, you'll find "bend rather than break", "ease of adaptation or offering many options", and (my person favorite) "the willingness to adjust one's thinking or behavior." Doesn't that last definition sound suspiciously like science and knowledge? It's no coincidence, and I'm not just playing word games. I'm pointing out the underlying truth that things must change. It's the third law of thermodynamics.

Any resilient system will necessarily resist ossification.

What are these quotes, by the way? Don't worry about it! I just made them up. If I used twitter, these are the kinds of one-liners I would tweet. But I don't, so I don't. You have to read the longform, instead.

I have argued so far that the concept of "stability" has something of a duality. There's the notion that something is stable because it resists external change, and the notion that something is stable because it resists bugs; it resists stagnation. An example of the former is a strong interface stability guarantee from a library. An example of the latter is keeping your browser and OS up-to-date to fix bugs, known security vulnerabilities, and just to acquire new features. Or stated another way, the "it's not a bug, it's a feature" line of thinking is the bug.

Lock yourself in

Having set the stage for stability as a means to lock-in a design or set it in stone, let's dive in to that concept a bit more. There are some examples in package managers where "locking in" your dependency list is a normal best-practice. npm has a package-lock file and a ci command to install packages from it. Cargo has a Cargo.lock file and installs packages from it by default (top-level only; lock files in dependencies are ignored). go.mod files have the same basic use.

Aside (added May 2024):

Nix takes package manager version locking to an extreme. My very broad understanding is that it uses hashes for package references in a similar way that git uses hashes for commit references. Meaning that it will scale well and be resilient, exactly the properties that we are looking for here!

I bring this up because versioning dependencies has been converging on this concept of locking the precise version number of every package in the tree within many major package managers. And it's no surprise, because this is one of the best tools for providing repeatable builds and testing. Semantic versioning plays right into that, as I covered earlier.

There's just one problem. This is not the kind of "locking in" that I allude to in the title. I'm talking about locking yourself into a specific design that is hard to change later. I'm talking about being that decisionmaker that owns the responsibility for putting a crappy middlebox on the Internet without ever updating it to support newer protocol revisions (or newer protocols, period). I'm talking about tying your hands by offering a stable API that just plain sucks. No, I take that back. I'm talking about tying your users' hands. Because it's not the author that is locking themselves into a contract. They are locking their users into a contract.

You may lock yourself in at your own peril, but you lock everyone else in with you at their peril.

This is something I take personal grievance with. Development philosophies like "worse is better" means that everything I use on a daily basis is figuratively worse than it could be otherwise. The Internet is held together with bubblegum and string. Wirth's law is real, and it threatens our very existence. I'll admit, using Wirth's law to criticize Bitcoin is a bit cheeky. But sometimes you just have to be cheeky.

I've rambled on enough, and linked to about a dozen articles that you should read. I'll just leave you with one last thought: Change or die.

Mutable statics have scary superpowers! Do not use them

In Rust, it is well-known that static mut is dangerous. The docs for the static keyword have this to say about mutable statics:

If a static item is declared with the mut keyword, then it is allowed to be modified by the program. However, accessing mutable statics can cause undefined behavior in a number of ways, for example due to data races in a multithreaded context. As such, all accesses to mutable statics require an unsafe block.

This makes perfect sense, so far. But my program is single threaded. I know it is single threaded because it runs on bare metal with no_std. I also know that I will need global state to manage access to memory-mapped I/O, and the global state needs to be mutable so that I can replace it at runtime. I can do this with static mut because I do not have to worry about data races with multiple threads right?

Well, that's the thought I initially had, anyway. And I believe I am not alone. But there is a lot of subtlety with static mut references that are problematic, even in a guaranteed single-threaded context. At first I believed it all has to do with the special 'static lifetime, but that is not the case. Rust By Example describes this lifetime as follows:

As a reference lifetime 'static indicates that the data pointed to by the reference lives for the entire lifetime of the running program. It can still be coerced to a shorter lifetime.

That also makes sense, and it's exactly what I want. In my case, the memory-mapped I/O that I wanted to control was a serial port. The primary goal was to reproduce something akin to standard I/O with println! and dbg! macros like those provided by the Rust standard library. A secondary goal was to make the sink configurable at runtime. This would allow the software to run on hardware with different serial interfaces, for example, without recompiling it for each distinct device.

The platform I was targeting is the Nintendo 64. Which, as you might guess, does not have your good old RS-232 serial port. It has a slot for game cartridges, and an expansion bus on the bottom which is basically just another cartridge port. It also has some controller ports on the front. Nothing really stands out as a plug-and-play means to get a serial console interface. As luck would have it, there are development cartridges that have an FTDI chip and USB port on them for exactly this purpose! The bad news is that each dev cart has its very own custom MMIO interface, and none of them are compatible.

Going back in time a bit, there are also debug interfaces on the original development hardware used in the 90's (and for a few years in the early 2000's) which use either parallel or SCSI ports. Some emulators support these interfaces because games (especially prototypes) actually print interesting things to them.

So we're in a situation where we want to write software for a platform (N64) that has a number of different serial interfaces available, but we don't want to build 3 or 4 different executables and put the burden on the user to figure out which one they need. And so here we are; we make the serial I/O implementation configurable at runtime. It can detect which interface is available, and that's what it uses. No sweat!

The core:fmt::Write trait

We want to use core::fmt::Write to make the implementation generic. This trait provides some useful methods for implementing the macros, and the core formatting machinery takes care of the details I don't want to think about. There are far better designs than what I came up with, but the initial implementation of my runtime-configurable I/O API would store a trait object (implementing fmt::Write) in a lazy_static. And the macros just needed a way to get a mutable reference to it. That's &'static mut dyn fmt::Write, mind you. But, I thought to myself, if I just hide all that inside the macro, nothing bad can happen!

What I didn't realize is that static mut has superpowers. I'll try my best to explain what these superpowers are, but please bear with me! I do not fully understand why these superpowers exist or what they are fully capable of. What I do know is that you never, ever, under any circumstance, want to anger them.

The 'static lifetime means that it lives forever (as far as the program is concerned) and making a static mutable means that you can only touch it in unsafe code, because threads and data races, right? Wrong! You need unsafe because the superpowers of static mut allow you to introduce instant Undefined Behavior. To explain, let's revisit what &mut means in the first place.

It's true that &mut allows you to change the contents of the thing behind it, but that's really just a side effect of the real purpose of &mut; it's an exclusive (unique) borrow. There can be only one.

All jokes aside, this is an invariant that you cannot break. Creating mutable aliases is instantly Undefined Behavior. Invoking Undefined Behavior gives the compiler license to do anything to your code. We really only want the compiler to do what we want it to do with our code, so let's just agree to avoid UB at all costs. To really nail this one down, consider what the Nomicon has to say about transmute:

• Transmuting an & to &mut is UB.
• Transmuting an & to &mut is always UB.
• No you can't do it.
• No you're not special.

There is some subtlety here that I won't go into, but the core of the issue is that this creates mutable aliases. Don't do it!

A starting point

There are some things that you will naturally begin to intuit while writing Rust after you have been using it for a while. It's helpful to recognize things that will cause the borrow checker to reject your code, for example. A quick internal borrow check in your brain can save a few seconds (or minutes, in extreme cases) of compile time just to be provided with an error message.

Let's begin with this bare minimum State struct:

#[derive(Copy, Clone, Debug)]
struct State {
    x: i32,
}

impl State {
    const fn new() -> Self {
        Self { x: 0 }
    }

    fn get_mut(&mut self) -> &mut i32 {
        &mut self.x
    }
}

There is nothing really special here, so far. You can gain a mutable reference to the x field if you have mutable access to the struct. Standard stuff, so far. If you attempted to write code like the following, it might make you pause as your internal borrow checker raises a red flag:

fn main() {
    let mut state = State::new();

    let a = state.get_mut();
    let b = state.get_mut();

    *a = 42;

    println!("I have two i32s: {}, {}", a, b);
}

Importantly, it isn't the explicit dereference (*a = 42;) that trips the borrow checker, here. It's the println! macro. And the error message illustrates this perfectly:

error[E0499]: cannot borrow `state` as mutable more than once at a time
  --> src/main.rs:22:13
   |
21 |     let a = state.get_mut();
   |             ----- first mutable borrow occurs here
22 |     let b = state.get_mut();
   |             ^^^^^ second mutable borrow occurs here
23 | 
24 |     println!("I have two i32s: {}, {}", a, b);
   |                                         - first borrow later used here

This mutable aliasing issue is something I've become more aware of over time, so it's kind of silly for me to write examples like this. The demonstration does help prove my point about superpowers though.

Let's make it static

Using the same State struct from above, let's make some minor adjustments to the program that uses it, so that State becomes static:

static STATE: State = State::new();

fn main() {
    let a = STATE.get_mut();

    *a = 42;
    
    println!("I have one i32: {}", a);
}

Your internal borrow checker should raise another red flag, here. We made the state static, but not mutable, so we cannot compile this code:

error[E0596]: cannot borrow immutable static item `STATE` as mutable
  --> src/main.rs:20:17
   |
20 |         let a = STATE.get_mut();
   |                 ^^^^^ cannot borrow as mutable

Everything is as expected so far. We need to make the static state mutable so that we can borrow it mutably (exclusively, remember). And that also means we need to use the unsafe keyword. We're about to find out why it's actually unsafe. :)

static mut STATE: State = State::new();

fn main() {
    unsafe {
        let a = STATE.get_mut();

        *a = 42;
    
        println!("I have one i32: {}", a);
    }
}

Well, that's not so bad! My internal borrow checker can't spot anything wrong with that. And neither can the compiler's (much better) borrow checker:

I have one i32: 42

But now I want to get spicy! 🔥 I want two exclusive references to the same memory. Or maybe I don't, but it accidentally happens anyway.

static mut STATE: State = State::new();

fn main() {
    unsafe {
        let a = STATE.get_mut();
        let b = STATE.get_mut();

        *a = 42;
    
        println!("I have two i32s: {}, {}", a, b);
    }
}

My internal borrow checker says this cannot work, just like it didn't work before. And the Rust borrow checker is way better than mine, so surely it won't allow this!

I have two i32s: 42, 42

ಠ_ಠ

(╯°□°）╯︵ ┻━┻

To be fair, trying to run this code in miri does detect the Undefined Behavior. This is a very powerful tool, along with the sanitizers. Always use these when you are writing unsafe code! No exceptions. This is a huge relief, or at the very least it's better than nothing. Here's what miri finds:

error: Undefined Behavior: no item granting write access to tag <2843> at alloc1 found in borrow stack.
  --> src/main.rs:23:9
   |
23 |         *a = 42;
   |         ^^^^^^^ no item granting write access to tag <2843> at alloc1 found in borrow stack.

Still, you may be caught off guard by this. Don't worry, it's a perfectly normal reaction. We didn't fundamentally change the State implementation, and yet putting it into a static mut changes its semantics. We might be given a hint about why this is if we desugar the get_mut method's function signature:

fn get_mut(&'static mut self) -> &'static mut i32

Go ahead, try it out! It compiles just the same. The static lifetime was simply elided. But wait, what if we keep this 'static lifetime and go back to the safe program code that puts State on the stack?

error[E0597]: `state` does not live long enough
  --> src/main.rs:21:13
   |
21 |     let a = state.get_mut();
   |             ^^^^^----------
   |             |
   |             borrowed value does not live long enough
   |             argument requires that `state` is borrowed for `'static`
...
31 | }
   | - `state` dropped here while still borrowed

So it is true that the 'static lifetime is special. We cannot borrow State for 'static because State lives on the stack. We would have to move it to the heap and leak the reference to make it live for 'static if we wanted to call the method with this lifetime bound. However, we cannot create mutable aliases in safe code with leaked boxes because they are not static.

In the case of static mut, we're allowed to borrow with a 'static lifetime, but we're also allowed to exclusively borrow multiple times. If you could only exclusively borrow a static mut once, it would make the feature useless for many valid cases. And this is the tricky part that, at least in my mind, makes static mut such a nightmare. You don't have a choice but to ensure the invariant that mutable aliasing does not occur, but the compiler won't prevent you from doing so. It is particularly insidious when your safe interface allows reentrancy, and that's how I hit this problem. Nested dbg! macro calls are pretty common.

This is the superpower of static mut.

How did I fix this in my configurable serial I/O?

Well, I'm not entirely sure that I did, to be honest! But I did find a way to get my macros to receive an exclusive borrow with a lifetime shorter than 'static. And I did that by not borrowing a static mut. Instead I have statics with interior mutability (STDOUT and STDERR). I opted to use spin::Mutex<RefCell<T>> to keep my uses of the unsafe keyword to a minimum, but it's also possible to store state with interior mutability in a static UnsafeCell. This is the recommended replacement according to Consider deprecation of UB-happy `static mut` · Issue #53639 · rust-lang/rust (github.com)

I also had to jump through hoops to get trait objects working without Box, but that was another rabbit hole entirely. In the end, I learned a valuable lesson about static mut and how bad it is for your health (and mine!) There are ways to make it work that are trivially safe, but reading through that GitHub ticket ought to ruffle your feathers, even if I haven't already convinced you that static mut probably should not be used.

The Rust compiler isn't slow; we are.

This might be a bit of an unpopular opinion, so the clickbait title should be appropriate!

I've been learning and using Rust for nearly four and a half years, since version 1.6. That's a good amount of time to become familiar with some of the troubles of the language, the tooling, and the ecosystem. But this is a slightly different story, this is a dissent to a common criticism that the compiler is slow. In this discussion, I will argue that these claims are misleading at best.

rustc compiling a non-trivial application.

There is plenty of evidence that suggests rustc is slow relative to compilers for other languages; Go can compile applications with similar complexity in 1/5 the time, and clang or gcc can do the same for applications written in C. It might be acceptable to justify that the Rust macro and type systems are significantly more complex than what is usually written in either Go or C, and that it warrants much heavier lifting by the Rust compiler. While there is some truth in that sentiment, I believe there is more to it than that. I believe it's more of a human problem than an algorithmic one.

Tiny MCU 3D Renderer Part 10: Benchmarking, profiling, and optimizations

The reason I have been so quiet recently is because I've been working on a very dry subject; testing. Unit tests are probably the most uninteresting thing I do on a daily basis as a software architect. Create some fixtures, write some assertions, run the test suite, fix any issues, repeat. It makes me cringe, just thinking about it.

But here I am, blogging about how I've been writing tests in my fun personal project, too! I am taking this project very seriously, so tests are a necessary evil. I haven't checked coverage yet (because I hit this bug with tarpaulin vs serde_derive), but it's probably somewhere around 70~80%.

Update 2017-11-05: serde_derive is only used in the app. The lib doesn't have this dependency, so I can run coverage on the lib separately. The result is even better than I imagined: 85.14% coverage, 573/673 lines covered

Surprise! There are only 26 tests (ignoring the 4 benchmark tests), how can coverage to be so high? This is one of the most interesting things about 3D rasterization; it doesn't require a whole lot of code! To be honest, though, I'm only exposing the bare minimum set of APIs; for example there's no support in the fixed function pipeline for stencil/scissor, alpha blending, etc.

Tiny MCU 3D Renderer Part 9: Bug fixes, and first shader tests

It's shader time! I finished the simple sunbeam, which I think looks quite nice in motion.



It could use a little work on the gradient, and some extra geometry wouldn't hurt. There are four sunbeams in this test scene. Each is just a plane (two triangles). I'm thinking of splitting the plane into thirds (vertically) so I can shape it more into a semi-circle, instead of just laying flat in the background. That might give it some depth, and help combat the problem with the right-most sunbeam looking so thin.

Tiny MCU 3D Renderer Part 8: Programmable Pipeline and Asset Pipeline

Oh hey, look at that! I changed the CSS on my blog a smidge. It's worth mentioning, anyway. Say hello to the Blipjoy Invader! You can just make it out on the left side, there. (Depending on your screen resolution, it might be behind the post content, whoops!)

I also finalized the programmable pipeline on the 3D renderer, thanks to @vitalyd over at the Rust user's forum for the hint I needed to push me in the right direction. The API isn't exactly what I had in mind, but it's certainly reasonable.

This is what I built yesterday. The model on the left is drawn with our very familiar Gouraud shader with interleave dithering on a four-color gradient. This is what we've seen exclusively in screenshots to this point. On the right is something new! A much simplified shader that renders something like a cartoon, aka cel shading minus edge detection. Each model rotates in opposite directions for funsies.

Code from the last article will be referenced below.

Tiny MCU 3D Renderer Part 7: Generics and Traits, oh my!

If you've been following my blog, you'll know that I've been writing a 3D renderer in Rust. This is my first real experience using the language. I dabbled a bit in Rust on a project in January 2016, where I was challenged by lifetime annotations, and gave up. This time around, I've managed to write a complete software renderer with all the bells and whistles of a modern shader model, without the need for a single lifetime annotation. And until just recently, without defining a single Trait or Generic.

Earlier articles in this blog series have focused exclusively on the renderer from an end user's point of view. In other words, trying to make it attractive to the everyday gamer. In this episode, I want to describe in detail one of the issues that I have been struggling with in the code design, and how I've been approaching a solution. This is by no means the "right way" to handle this, or even similar situations. I just want to provide some info for anyone who happens upon the right magical incantation in The Googlies, and ends up reading this.

(Edit 2017-10-04: The illustration below was originally flipped vertically. It now shows the correct scanning direction.)

Triangle rasterization scans pixels in the target buffer from left-to-right and bottom-to-top

So let's start from the middle, and work our way out. Above is a simplified representation of the rasterization step of the renderer. The details prior to, and those that follow rasterization can be ignored for the time being.

This image represents a zoomed-in detail of a frame buffer (or any render target) as a 2D triangle is being rasterized. This occurs after the perspective division, so the Z coordinate can simply be dropped for rasterization; the Z component is used later for depth testing.

Tiny MCU 3D Renderer Part 6: Camera animation and display scaling

Yesterday I finally got around to adding some simple animations. The app was always rendering at 60 fps, but the image was static because there was no animation. That's why I've only been posting PNG images of progress so far. But now I can do this:


This was my first ever foray into quaternions! And I must admit, learning about quaternions suuuuuuucks. Surprisingly, this is one area where I would actually recommend developers keep quaternions as a mysterious black box, though an essential part of their repertoire. Every academic writing you will find about quaternions is deep in imaginary number territory (which we all know is impossible to represent on a computer). The important point is that the imaginary numbers can be optimized out, so having them in the first place is completely stupid, but I digress. Ok, ok, imaginary numbers help make sense of the derivatives... So what? It still sucks. And it's still pointless in the context of 3D graphics.

Tiny MCU 3D Renderer Part 5: Aspect Ratio and Field of View

I had a long week on vacation, and was able to do a little bit of coding almost every night. There was a lot of time spent doing touristy things, so my coding opportunities were limited. I had a good solid 4 hours of nothing but coding time on the plane, though! Both ways.

On my departure flight, I managed to finally fix the aspect ratio (as far as I can tell). This was just a matter of adjusting the projection matrix to use the correct aspect ratio for non-square pixels. On my return flight, I finished almost all of the refactoring for the new Shader API, and finally completed it from the comfort of my own couch.

It shouldn't look too much different from the previous screenshot. There are a few obvious differences if you look closer, though.

Quick update, progress report, current plans

This weekend I was distracted by a well-intentioned good friend of mine who suggested solving a chess puzzle described as "deceptively simple". Unfortunately, the article is criminally misleading. Therein it is claimed that computers cannot "solve the conundrum quickly and efficiently". The article is misleading because it is in fact trivial to solve the puzzle in linear time with constant space complexity.

Solving the puzzle is not the challenge alluded to. The challenge is that given any starting position with queens already placed, find a valid solution by adding more queens. A notable related problem is enumerating and counting all valid solutions. To date, it has been shown by brute force that a 27x27 chessboard has over 243 quadrillion solutions; removing all symmetrical solutions results in about 29 quadrillion. The brute force work took about 7 years with a massively parallel array of custom hardware (written to FPGAs).

To provide some context to the size of the numbers involved, it would take a modern CPU (single-core at 4.5 GHz) about 4 years just to increment a counter as fast as possible from 0 to 29 quadrillion. That is the time estimate just for the work involving the counter; nothing more. It's also hopelessly optimistic, since that assumes you already have a list of all 29 quadrillion solutions, or that there are zero false positives or any instances of wasted effort.

Tiny MCU 3D Renderer Part 4: Gouraud Shading

Today, it's interpolating normals to render smooth lighting. That's right; Gouraud Shading in full effect. Two screenshots to start with; first is a view with diffuse disabled, to show the full effect of the shading. Followed by fully textured.

Surprising that the texture is so dark. But it is what it is. I think this test model has just about reached the end of its usefulness for the project. There's just one duty left for it to serve. Remember those gradients at the bottom of the image that were added in Part 2? It's time to put our friend here through some post-processing!

Tiny MCU 3D Renderer Part 3: Textures and Perspective

I was surprised by how easy it was to interpolate over the texture coordinates, given the barycentric coordinate space. I have more boilerplate code to convert the mesh vertices into cgmath vectors than there is code to interpolate the triangles! I'll refactor it all away after the renderer's features begin to stabilize. With a little gamma and luminance love, I now have nearest-neighbor texture mapping:

The gamma correction was crucial, since this texture went through two separate processing passes; first, I dropped all chrominance information from the RGB leaving just the relative brightness; and second the global illumination was applied to the texture mapped geometry as you might imagine. To get it working right, the RGB components are transformed from Gamma Space to Linear Space prior to the luminance transformation. Then the texture stays in Linear Space until after the final illumination pass. The pixels are transformed back to Gamma Space as they are written to the frame buffer.