Scanning, paths, hardened keys, derived addresses…these are just a smattering of the ‘fun’ nomenclature that awaits you if you dip your toe in the murky pond that is Bitcoin extended public keys. As with anything crypto related, hold on loosely, and I’ll attempt to show you around the world of extended public keys in a way that helps you both maintain your sanity and your funds…

If you want to open up a new Bitcoin wallet today, chances are it will use what’s called an extended public key. This bakes in behaviours that might at first seem strange and confusing. Every time you want to receive some Bitcoin, and you ask your wallet for your receiving address, a brand new address is generated. If you ask for your extended public key, you might not be able to find your own transactions. Wallet providers seem to have a ‘scan’ utility to look for BTC connected to your address. This seems bizarre – how can an account full of money seemingly not know where the money is? Worst of all, research into what’s going on is hindered by the appearance of words and acronyms like ‘xpub’, ‘segwit’, ‘BIP’ and the truly arcane ‘P2WPKH in P2SH’.

So let’s figure it out!

Full disclaimer: I’m not a mathematician, and I will use the term ‘cryptograpic magic’ liberally throughout this article to describe any processes where I can tell you what something does, but not how it works!

The Basics

At the most fundamental level, if you wish to receive some Bitcoin, you must have an address at which to receive it. As ‘Learn Me a Bitcoin’ (which is probably the best explanation of all things Bitcoin out there) explains, this can be thought of as analogous to a bank account number – at first. This metaphor is now pretty dated, but we’ll stick with it for now (until we see the reasons why it’s not a great comparison). Once created, an address is publicly available, and derived from something called a public key (which is just a longer, harder to grok version of the same thing). A corresponding private key exists, which is analogous to your bank account password (but a far more extreme version, that overrides any and all security. No secret questions and answers here).

While it’s perfectly possible to use just this one address to send/receive BTC, this has some pretty hefty privacy and security implications. As soon as you send this address to someone (which you have to do to receive money), they can use this address to look up your entire transaction history. There’s also the problem of double-spending attacks – a vulnerability which specifically targets re-used addresses.

The answer to these problems came in the form of wallets – a sort of key-chain of public/private keys (and the corresponding addresses) that you can use to access and generate new addresses. Here’s where we should retire the ‘address as a bank account number’ metaphor, and replace it with the far better ‘address as an invoice’ metaphor. When you want to receive some money, you use the wallet to generate a new address/invoice. When that’s paid, you never use that address/invoice again. This even works in reverse; if you want to pay someone, due to the way Bitcoin works, the entire balance of that address is spent, and then your unspent transaction outputs (UTXOs) are given back to you. Your wallet will put these funds in another brand new address – so all the addresses involved are only used once.

If that’s a little hard to grasp, here’s something concrete to visualise. Imagine you want to buy a $0.65 item in a store, and you have 100 cents. Using the wallet model, you’re going to reach into your left pocket and hand over all 100 cents to the shopkeeper. After some frowning and counting (and taking an extra 10 cents for taxes), she’ll give you back a quarter, which you’ll now put in your right pocket. You’ll never use your left pocket again. Subsequently, when you want to buy something worth 15 cents, you will hand over your quarter, get back your change minus the tax, and put it in your front pocket.

While this all works very well for maintaining your privacy and keeping things organised, this creates a new problem. To maintain access to all your addresses, you have to keep the corresponding list of private keys. That list gets longer and more unmanageable over time – and pretty quickly too, if you’re party to a lot of transactions. To back it up, you’d have to back up the entire list every single time a payment was made or received.

What’s the solution here? Enter BIP32 and the HD wallet.

Hierarchical Deterministic Wallets

Bitcoin Improvement Proposals (or BIPs) are proposals made and accepted/rejected by the Bitcoin community. Some of these affect the Bitcoin software and network itself, and some of them are concerned with implementing standards around that network. The Bitcoin community is always trying to be better, more secure and maintain more privacy for users, and BIPs provide a forum for putting ideas out there and implementing them.

BIP32 proposed the idea of hierarchical deterministic wallets. Instead of generating new private-public key pairs for every new transaction, only one parent public/private key pair is generated. Using cryptographic magic and the parent public key, billions of new child public keys can be generated without giving away the parent private key. Simple deterministic wallets which can do something similar, but that method requires the parent private key to generate the new child public keys (and we want to keep the private key private at all costs).

Derivation paths (how HD wallets work)

The best way to visualize this is like a tree, with the private/public key pair (or seed) at the root. Over four billion private/public key pairs can be generated from that seed, which all lie at different levels within the tree. Let’s say you want the key pair that lies at the first branch of the first branch of the first branch – in conventional notation, this would be the key pair at m/1/1/1. M is the seed (or master seed), and the slash separated number combination represents the path you took to arrive at the key pair. This is known as the derivation path – the path you have to follow to get to the key pair.

Another BIP came along (BIP44), and proposed that the format of these paths within the wallet be standardized. For example, it proposed that all receiving addresses (addresses for the wallet to generate when you want an address to receive funds) should be derived at m/0/[here], and all ‘change’ addresses (addresses where your unspent transaction outputs are sent) should be derived at m/1/[here]. So for example, your first receiving address will be generated at derivation path m/0/0, your second address at m/0/1and your ‘nth’ address at m/0/n.

Are you still with me? Good! Let’s make it even more worryingly messy. Each private/public key pair that the seed generates can also be extended. You heard me. That means at a certain point on our tree, we could grow a brand new tree. So I guess you can think of things as more like an infinite fractal. Sorry.

There’s benefits to this though. All you need is one crypto wallet; and within that, you can have a wallet for personal expenses, a wallet for public trades, a wallet for doge coin… Using your one extended parent private key, you have access to all of them, and all of them are extended key pairs, meaning they can all generate fresh addresses.

There’s (just one more, I promise) an extra layer of magic called ‘hardened’ derivation paths. These can be thought of as a sort of one way valve. There could be a situation where an attacker had their hands on a non-hardened extended child private key (let’s say the one you were using for personal expenses). Because they know the non-hardened extended child private key, they have access to spend all funds tied to keys connected to that child. But in addition, if they ever got your parent extended public key (which, after all, you probably haven’t treated with the same level of caution as your private key) they could figure out your parent extended private key, and seize control of your whole wallet (and in doing so, steal your doge coin).

On the other hand, if the extended child private key they managed to get ahold of was hardened, it would be impossible for them to access the rest of the wallet; even if they found out your extended parent public key. Through cryptographic magic, it’s impossible to get to your extended parent private key. Your personal expenses funds are gone, but your precious doge coin would be safe.

And now, to tie it all together – the eponymous extended public key.

Extended Public Key

To recap the above: Using an extended public key, you can generate billions of fresh receiving addresses ( 4,294,967,296 to be exact). This maintains your privacy, security and sanity (as there’s only ever going to be one private key that can access the funds connected to those addresses.) Even better, while your extended public key can be used to generate addresses (and therefore be used with any software you write, and any software you trust) it categorically cannot be used to access any of your funds.

It’s worth mentioning that you don’t want to spread the extended public key around that much. While it can’t be used to spent your funds, the triumph of privacy that the extended public key achieves is rendered void if your extended public key is, well, made fully public. Anyone with it would be able to see all your transactions.

However, you don’t need to treat the extended public key with the same level of caution as your private key. If you want to write some software to generate addresses for you, you can load up your extended public key, safe in the knowledge that if there’s ever a leak or a hack – your money is safe.

It’s also worth mentioning that you can’t send money directly to an extended public key – it can just be used to generate addresses to send money to.

Extended public keys, helpfully, have the word pub in them. Anytime you see xpub, ypub or zpub, you’re looking at a bitcoin extended public key. Indeed, if you look in the right place in your wallet, you can find generally your extended public key. This is the piece of information that the wallet is using to generate all your Bitcoin addresses.

However, there’s one extra layer of complexity on top of this – and that’s the kind of address your extended public key generates.

Legacy, Segwit and Native Segwit

When looking at Bitcoin addresses (the ones you can send money to), you may have noticed that they start with either a 1, a 3 or bc1. This refers to the way in which the Bitcoin is transacted to that address, but for the sake of simplicity, let’s call these address types. 1 is Legacy – the oldest type of Bitcoin address around. These addresses work just fine for transactions, but fees will be higher than the other two types. This is because the newer two types of address – Segwit (3) and Native Segwit(bc1) – have been engineered to use less space on the blockchain.

Legacy addresses use pay to public key hash (P2PKH) which is fairly straightforward – payment is sent to the hash of your public key. Segwit uses pay to script hashing, which essentially means that there is a piece of code that must be satisfied (a script) before the Bitcoin in a transaction is actually transferred. Finally, native segwit addresses use pay to witness public key hash, which means that the script that witnesses the transaction is moved out of the transaction body and placed in the ‘witness’ body. Explaining this is a little beyond the scope of this article, but if you wish to learn more, I’ve included some learning links below!

So what does this have to do with extended public keys? Extended public keys can only generate one type of address – legacy, segwit or native segwit. The type that they can generate is determined by the first four characters in the extended public key – xpub, ypub and zpub.

Address Type	Address starting character format	Transaction method	Main net Extended Public Key identifier	Test net Extended Public Key identifier
Legacy	1	P2PKH	xpub	tpub
Segwit	3	P2SH	ypub	upub
Native Segwit	bc1q	P2WPKH	zpub	vpub

A breakdown of the different address types

‘Segwit’ is short for ‘Segregated Witness’. Essentially, data about the transaction (but not the transaction itself) is stored separately from the transaction data, still on the blockchain, but in a ‘witness’ block. This is what allows these address types to have smaller fees. The older legacy method simply put all this information in with the transaction and pointed everything at a hash of the public key: a method called pay to public key hash (P2PKH). The next iteration pointed all the information towards a hash of a script which splits up the transaction _data_ and the data _about_ the transaction, and saves it in two different places on the blockchain: a method called (you guessed it) pay to script hash. This allows for backward compatibility and smaller fees. Finally, native segwit splits things up right off the bat, and is called pay to witness public key hash (P2WPKH). This means tiny fees – but no backward compatibility.

So why can I only find my xpub?

There are some wallet providers (which I will not mention) that will only ever provide the xpub, even when the wallet in question is a native_segwit wallet. This is useless at first blush – you’ll only be able to use a xpub address to generate legacy addresses, and any attempt to find your native_segwit addresses through a service like blockexplorer.com fails. However, you can still use this extended public key – it just needs some massaging first.

xpub is the oldest format of extended public key address – that’s where the e(‘x’)tended comes from! In a lot of applications, it’s become shorthand to mean extended public key, rather than ‘extended public key that only generates legacy addresses’. Therefore, even if you select segwit or native_segwit when creating a new wallet, you may only be given an xpub when asking for your extended public key.

But don’t fret – there are wonderful tools that exist in the forms of libraries, software and websites to help you get your keys into the correct format. I’ve linked a website below that you can use, as well as link to the javascript source code if you want to do it yourself. Once you’ve successfully converted your xpub (to a ypub for segwit, or a zpub for native_segwit) you’ll be good to go on generating addresses or looking for your own used addresses to your heart’s content.

And that’s it! You now laugh in the face of leaks exposing your extended child private keys, because you used hardened derivation paths! The xpub your wallet provider gave you simply needed to be converted to a zpub to find the native_segwit addresses your money was sent to! Best of all, no one who ever sends money to you can see has a window into your finances, because you are using a Bitcoin extended public key!

Sources and resources

Bitcoin Improvement Proposals

Bitcoin education

xPub/HD wallet education

Thank you for following along while we explain the mystery of the extended public key.

We're building an AI-powered Product Operations Cloud, leveraging AI in almost every aspect of the software delivery lifecycle. Want to test drive it with us? Join the ProdOps party at ProdOps.ai.

Bitcoin Cryptocurrency Emerging Tech Stuart Page

The Bitcoin Extended Public Key: Explaining the Mystery