Okay, so yes I am going to break a couple of rules here. But really only the first two…
Types are literally everywhere; they are the only way digital computing actually functions and does what we expect. Yet they are entirely a construct of our own imaginations. If you are willing I’d love to take you on a little journey involving a “single serving friend,” a “puzzle” and a healthy dose of editorialization.
Jack’s Budding Self-Awareness
Imagine you are headed home from a business trip. You wake up from a nap and notice someone different sitting next to you; they start reading the safety card which you take as an opening for a conversation. The topic naturally gets around to careers. They tell you they are a software engineer. Right before you land, you work up the courage to ask them about programming. They give you their business card; you flip it over and see a bunch of garbage written on the back of it.
010011010101000001010011001101110000000100000000000000000000000001000111000000000101001100001001001001111101000100111001011001111100000001000101110110011001000100100001010000001000001011100010001100011101011011010111001011101000110100000001010101000110110101110001011010100001010010100010101100001111010111010100101110111000111111100010010000001000100111001001000010010111001110000001100100110110101000000010010100101110000111110100101110110011001110101111110100110111100101000001010000111001000010001001Before you can ask what the hell it is supposed to be they get up and go sit somewhere else on the plane.
Jack’s Medulla Oblongata
Exhausted, you head back to your apartment. Instead, you discover a plot device – your apartment has exploded.
Through your shock, you vaguely hear someone asking if you have anyone you can call. For some reason, you take the card out of your pocket and call that single serving friend you met briefly on the plane. Your nerves could really use a drink, so you end up deciding to meet at a local bar. Somehow, you manage to forget about your smoking crater of an apartment, and your curiosity starts to take over again. You ask about the gobbledegook on the back of the business card. The stranger laughs and says:
I’ll give you a hint. It is the beginning of an accounting ledger – well more of a log really – but ultimately just financial data. Doesn’t look like much of anything unless you know how to break it down and put it back together. You can figure it out if you know the correct assumptions about the data. Let’s call those assumptions types.
Let’s start at the root with the raw data. After all it’s only after we’ve lost everything that we’re free to do anything. In any digital computer your atom is the bit. This is about as simple as things can ever get. A bit is a single “binary” digit either a 1 or a 0, that’s it end of story.
Computers are composed of a vast multitude of switches or gates which fundamentally operate on electricity being present or not. Thus you have two states you can represent; on/off, 1/0, yes/no whatever you choose to call them that is all you really have. Maybe interestingly, maybe not, this is also one of the few places/times where we “type” something by making it less abstract. Maybe it is really a conversion more than typing but I still think that is mildly interesting.
Okay, I may have lied a little bit and there is another subtle bit of “typing” going on here however since that is mostly a hardware thing we’ll pretend it doesn’t exist. This likely isn’t really enlightening yet but it maybe raises a more general WTF type question in your head. The trick lies in applying patterns and abstract meaning to that sea of points. We can revisit our model as we work to help us figure out what needs to happen next. Think of me as your reference material. I know where we want to go and I can give you some vague directions after each step.
He takes a cocktail napkin and writes a bunch of 1s and 0s down again, but this time, there’s a discernible pattern to them
01001101 01010000 01010011 00110111 00000001 00000000 00000000 00000000
01000111 00000000 01010011 00001001 00100111 11010001 00111001 01100111
11000000 01000101 11011001 10010001 00100001 01000000 10000010 11100010
00110001 11010110 11010111 00101110 10001101 00000001 01010100 01101101
01110001 01101010 00010100 10100010 10110000 11110101 11010100 10111011
10001111 11100010 01000000 10001001 11001001 00001001 01110011 10000001
10010011 01101010 00000010 01010010 11100001 11110100 10111011 00110011
10101111 11010011 01111001 01000001 01000011 10010000 10001001Sometimes a little bit of structure can go a long way. We’ve really only applied two concepts here but you will find they are pretty fundamental when it comes to low level data representations:
- Partitioned our bits into bytes
- Partitioned our bytes into “words”
So, we kind of have our first two “types”. A byte is a collection of 8 bits. In this case it represents a number from 0-255 (these are unsigned). We also have a word this usually reflects the maximum capacity of a CPU register (and consequently a memory address) on the computer in question. Word size can vary from computer to computer, these are 64-bit words.
This is the point where things can start to bridge the gap from machine to man. Looking at a gigantic string of binary can make your eyes bleed. However, adding a little bit of formatting can help you keep track of where you are in the data if nothing else. There is an else though! The fact that this formatting maps to something that happens in the computer hardware helps, too. We are starting to establish a bit of a common language with the computer!
Jack’s Nascent Clarity
Surprisingly that makes a bit of sense. Okay, so how do these groupings turn into a log/ledger? The obvious part is that there will be a list of records of some kind in here. What might not be obvious is that we are also expecting something called a header which tells us a bit about the data.
Given that information we can at this point determine that those bytes are storing two things:
- Header
- Content
Jack’s Complete Lack of Surprise
This means that like before we should be able to peform some kind of grouping/partitioning and at least see if this data really can represent a transaction log.
Header
If we want to partition our data into a Header and Content we need to know how to separate those two. Where does one end and the other begin? Your single serving friend grabs another cocktail napkin and at the top they write Header with the following information underneath:
- Total Length: 9 bytes
- Format:
<magic:4><version:1><records:4>- magic –
ascii - version –
u8 - records –
u32
- magic –
You look at it for a minute and ask: “Okay, but what does that mean really?”.
It means that we are expecting our header to be a total of 9 bytes. We’ve already seen that a byte is 8 bits which means that the first nine groups should make up the entirety of our header. Going one level deeper: our header is actually a compound type and is really three different values in sequence. The first piece of the header is 4 bytes in size and is what is sometimes referred to as a “Magic String”.
This is a marker that helps an application determine if it knows how to process this type of data. The next byte in the header is a version indicator, this could really mean anything but for our purposes it mostly just serves as a place holder.
Knowing it will be there and accounting for it helps us avoid looking in the wrong place for the next value. The final 4 bytes indicate the total number of records we should expect to find in this file. This would be super useful if we were going to do any kind of integrity checking etc.
Bear with me, there is a good reason for this. Let’s actually start from the 9th byte and work backwards. You probably wouldn’t do this in real life but I haven’t explained “strings” yet so let’s get the numbers out of the way first.
Records
So, starting at the “end” we have the record count. Going back up to our Header typing info we can see that the records value is something referred to as a u32. This means that those bits actually make up one 32-bit number:
00000000 00000000 00000000 01000111If we convert it from binary to decimal we discover that the header claims there are 71 records. After performing the conversion you comment that the records must be really small. Your companion informs you that there actually aren’t 71 records in the data. It needed to fit on a business card so quite a few have been omitted. Technically there is capacity in the count to reflect 4,294,967,295 records. Who really knows if this format was ever intended to hold that many records in a single log or if it was just originally written on a 32-bit machine…
Version
Okay, so that’s the records out of the way, next we have the version. Again going back to the typing info for the Header we only have one byte (8-bits) for the version, that should be a pretty easy one.
00000001Look at that we’ve got version 1!
Magic String
Now for the first/last piece of the Header the “Magic String”. Looking back to the Header typing we can see that it consists of 4 bytes (32-bits).
01001101 01010000 01010011 00110111“Magic String” implies that we are expecting letters rather than one or more numbers though. So, how exactly are we supposed to do that?
If you’ve played around with any cryptographic cyphers or even super simple decoder rings you probably have an idea. And you wouldn’t be wrong. There is an encoding called ASCII that is an implementation of that very concept. It assigns numbers to different “symbols” i.e. letters and other characters found on English language (American) keyboards (along with a few extras). So, this is easily our most abstract type so far. We are basically following this path: bit -> byte -> character/symbol.
So, if we look at an ASCII table we discover that the four bytes above could possibly be a representation of MPS7
Neat so we have applied some primitive “concrete” typing and now have a Header:
MPS7 1 71Jack’s Cold Sweat
Okay, so the first part is sorted and has undergone a rather radical transformation in the process. But the Content is still a mystery. What the hell is it exactly? How do you go about deciphering the information there? All you know right now is that there aren’t 71 records but really that just gives you an upper bound. So, you turn to your single serving friend again and before you can ask they start writing on another napkin:
Content
There are four possible records in the content section:
- Debit
- marker – 00000000
- length – 168 bits
- Credit
- marker – 00000001
- length – 168 bits
- Autopay start
- marker – 00000010
- length – 104 bits
- Autopay end
- marker – 00000011
- length – 104 bits
So, this is helpful and if we look at the start of the content it tells us that the first record should be a Debit.
00000000 01010011 00001001 00100111 11010001 00111001 01100111 01000111
11000000 01000101 11011001 10010001 00100001 01000000 10000010 11100010
00110001 11010110 11010111 00101110 10001101 00000001 01010100 01101101
01110001 01101010 00010100 10100010 10110000 11110101 11010100 10111011
10001111 11100010 01000000 10001001 11001001 00001001 01110011 10000001
10010011 01101010 00000010 01010010 11100001 11110100 10111011 00110011
10101111 11010011 01111001 01000001 01000011 10010000 10001001Jack’s Budding Serenity
Using our new Content typing information it looks like we have 3 records here:
Debit
00000000 01010011 00001001 00100111 11010001 00111001 01100111 01000111 11000000 01000101 11011001 10010001 00100001 01000000 10000010 11100010 00110001 11010110 11010111 00101110 10001101Credit
00000001 01010100 01101101 01110001 01101010 00010100 10100010 10110000 11110101 11010100 10111011 10001111 11100010 01000000 10001001 11001001 00001001 01110011 10000001 10010011 01101010Autopay start
00000010 01010010 11100001 11110100 10111011 00110011 10101111 11010011 01111001 01000001 01000011 10010000 10001001Finally, we almost have something useful put together. And for some levels curiosity this could be satisfactory but why stop at satisfactory? I’m sure there is more information about what a Debit, Credit and Autopay start are. Maybe there is an actual end to this tunnel. Right on cue your single serving friend is already working on a new napkin:
- Debit –
<timestamp:32-bits><account:64-bits><amount:64-bit float> - Credit –
<timestamp:32-bits><account:64-bits><amount:64-bit float> - Autopay start –
<timestamp:32-bits><account:64-bits> - Autopay stop –
<timestamp:32-bits><account:64-bits>
Okay, that seems sane, Debit and Credit have the same composition and so do Autopay start and Autopay stop. Given the “names” we’ve been given for those things that seems to make sense. So, let’s apply this new type information and see what we get.
Jack’s Blinding Clarity
Alright, using the our newest breakdown we have the following:
Debit
- timestamp –
01010011 00001001 00100111 11010001 - account –
00111001 01100111 01000111 11000000 01000101 11011001 10010001 00100001 - amount –
01000000 10000010 11100010 00110001 11010110 11010111 00101110 10001101
Credit
- timestamp –
01010100 01101101 01110001 01101010 - account –
00010100 10100010 10110000 11110101 11010100 10111011 10001111 11100010 - amount –
01000000 10001001 11001001 00001001 01110011 10000001 10010011 01101010
Autopay start
- timestamp –
01010010 11100001 11110100 10111011 - account –
00110011 10101111 11010011 01111001 01000001 01000011 10010000 10001001
For the most part these new details are the same as the majority of what we’ve been working with previously, integers of various “widths”. But there is something in the Debit and Credit that is new. The amount is a “64-bit float”. You look up at your single serving friend and ask what exactly that means. They explain that computers are not great at representing fractional numbers. In order to try and work around this there are some clever tricks you can use. They also mention that floating point isn’t exactly perfect and probably was not the best choice for a financial ledger.
The problem is that there are a lot of numbers you can’t accurately represent and when performing mathematical operations with floating point numbers the computer rounds (by very small amounts but it does round).
They also mention that the rest is up to you. They don’t actually have any further information about these final types. This means that the timestamp could maybe be a simple count of some amount of time that has elapsed since a specific origin (or epoch). Or it could be a representation of some form of date or other representation with more definition. They also point out that the account identifier could be a number or it could be some kind of string or it could be something else. However even without those details you can still determine quite a lot about the data and probably perform any kind of aggregation/sorting/filtering you might want to do.
Armed with this final bit of information you take another look at what you have so far. You can say for certain that none of these records were created at the same time. Which also implies that you could apply some sort of ordering to them if pressed. It is also clear that these three records are for different accounts. You don’t really need to kown what exactly the account identifier ultimately represents in order to “balance the books” so to speak. So, really the last refinement we can deterministically make is to figure out the amounts for the debit and credit.
If we use the IEEE 754 standard for computing our amounts we get:
- Credit: 604.2743355570870562587515451014041900634765625
- Debit: 825.129614841757984322612173855304718017578125
This is a great example of why floating point isn’t a great representation for financial data; the values encoded here are highly unlikely to have been the actual amounts.
Jack’s Insincere Apology
That was a surprising amount to figure out one Debit, one Credit and an Autopay start (whatever the hell that is). Hopefully you found some of it interesting, and if not well at least you scrolled all the way down here. The main point here was to provide evidence/support for the following claim:
Typing is Foundational
As we saw in the beginning the raw data was inscrutable – it could have meant anything. You may have also noticed that there really wasn’t anything an actual computer would do that helped us figure what what it represented. There is no way for us to reasonably work with computers without leveraging typing to some degree (punch cards are not a reasonable way to work with computers).
There is even more to typing than we explored here. This example was heavily focused on concrete typing. That is to say we were primarily focused on the “physical” layouts of things and applying definitions to raw data. Abstract typing is very much a thing too and maybe one that more developers are familiar with, to varying degrees. When you are working with more abstract types you are still describing the data but equally or sometimes more importantly you are describing behaviors of that data. What is it allowed to do, how does it change/behave in different scenarios if at all. For example the abstract side of what we worked on would likely set rules like:
- Math is allowed on the amounts for Credit and Debit records
- Autopay records must have a specific order of environments
- any number of other things
We did touch on a few things that were more on the abstract side. Our Magic String is a good example, that was really one step above the actual layout (width) of the data. The general Debit, Credit and Autopay internals sort of straddle the line a bit too in that we are attributing a domain meaning to parts of that layout. Things tend to get even more abstract the farther up you go. This can be powerful but also like any abstraction it can make other things much harder to do.