Cookbook

HBS2 Cookbook

HBS2 Cookbook

Assuming both hosts have hbs2-peer and it is running.

On host A:

A:$ hbs2 store yehatrmx.mp3
5sxd7TWeNGbKM5zxBg9acCP7MdbT92T7rN3LHyJB8KrA

A:$ hbs2 hash yehatrmx.mp3
Dtt3PKFuvyJtzWa46Bv8jP7atCqX3tuiqo6v3xf81irr

On host B:

B:$ hbs2-peer fetch -p 5sxd7TWeNGbKM5zxBg9acCP7MdbT92T7rN3LHyJB8KrA && \
    hbs2 cat 5sxd7TWeNGbKM5zxBg9acCP7MdbT92T7rN3LHyJB8KrA > yehatrmx.mp3

B:$ hbs2 hash yehatrmx.mp3
Dtt3PKFuvyJtzWa46Bv8jP7atCqX3tuiqo6v3xf81irr

What’s Going On Here

On host A, we store a file in hbs2 local storage using the hbs2 store command.

It generates a Merkle tree for the file yehatrmx.mp3. The hash returned by hbs2 store is the root hash of this Merkle tree:

hbs2 cat -H 5sxd7TWeNGbKM5zxBg9acCP7MdbT92T7rN3LHyJB8KrA

AZ57xyNxtEGZdFtMyZFiQBVbUAGsXAxPYah7WsjUZSBS
Eq72bbt9Nhoco1EfW5SJ9r9WBMSyjESd2cgpgC4k5wKz
6nHDUbtweNRykqBnb8etHpoxYM7Tquvt1FGfDWtMGS38
DMAt8FeLzFwMv8P9zTAdF1AkXyeVK6y3BramZuo7zhot
6uB6WAir6TPfqvv3CtVVAVPm2ityaXjBuXa6SsgVfmo6
...
J1XRxrDCWsLsoEUK5MMdakxb9sZzghrmxfBiqewKpc64

This particular Merkle tree contains a list of blocks, each of which is a piece of the original file:

hbs2-cli [iterate [fn 1 [println _1 space [hbs2:peer:storage:block:size _1 ] ]] \
[flatten [call:proc hbs2 cat -H 5sxd7TWeNGbKM5zxBg9acCP7MdbT92T7rN3LHyJB8KrA]]]

AZ57xyNxtEGZdFtMyZFiQBVbUAGsXAxPYah7WsjUZSBS 262144
Eq72bbt9Nhoco1EfW5SJ9r9WBMSyjESd2cgpgC4k5wKz 262144
...
J1XRxrDCWsLsoEUK5MMdakxb9sZzghrmxfBiqewKpc64 117156

As you can see, each block is 262144 bytes, except for the last one.

Each block is stored in hbs2 storage separately and may be accessed by its own hash. If we concatenate those blocks, we will get the original file as it was:

hbs2-cli [iterate [fn 1 [println [unwords hbs2 cat _1]  ]] \
[flatten [call:proc hbs2 cat -H 5sxd7TWeNGbKM5zxBg9acCP7MdbT92T7rN3LHyJB8KrA]]] | sh > out.bin

hbs2-cli [iterate println [hbs2:hash yehatrmx.mp3 out.bin]]
Dtt3PKFuvyJtzWa46Bv8jP7atCqX3tuiqo6v3xf81irr
Dtt3PKFuvyJtzWa46Bv8jP7atCqX3tuiqo6v3xf81irr

sha256sum yehatrmx.mp3 out.bin
7ab1adc70316b1d055af1902d69b30dc3af45d4e43dfde7c0028e5ab531aeadd  yehatrmx.mp3
7ab1adc70316b1d055af1902d69b30dc3af45d4e43dfde7c0028e5ab531aeadd  out.bin

Each of these blocks may be requested by any other peer that can access hosts A or B.

All blocks persist in local storage on each host until they are explicitly removed.

This example works only if Peer A and Peer B can access each other and are both online.

The file stored this way contains no metadata, so if you forget what it was, you will need to figure it out by its content.

The file stored this way is not encrypted—any other peer or person may fetch and access its content.

If this is not what you want, hbs2 provides solutions for these issues:

It can encrypt files (or any data).
It has multiple ways to notify other hosts about new content.
Peers can download content if they subscribe to a notification channel (or reference).
hbs2 can add metadata to files, such as HTTP access.
hbs2 can encrypt blocks for multiple users using an end-to-end group key (GK).

All peers/users with the key can read the original file, while others can only download and seed it.

These topics are explained in other manuals.

Get this site to work on localhost

Install hbs2, then make hbs2-peer running

hbs2-peer run

And on another terminal

hbs2-peer poke | grep http-port

http-port: 5000

hbs2-peer poll add 4X65y4YvUjRL2gtA9Ec3YDDP4bnxjTGhfjpoah96t3z1 lwwref 31

wait some time for hbs2-peer to fetch all data, and then

firefox http://localhost:5000/ref/4X65y4YvUjRL2gtA9Ec3YDDP4bnxjTGhfjpoah96t3z1

What’s going on here

We’re subscribing to reference 4X65y4YvUjRL2gtA9Ec3YDDP4bnxjTGhfjpoah96t3z1 of type lwwref and want to poll it one time in 31 minutes.

All lwwref updates are propagating by GOSSIP protocol, so you do not need to poll it frequently. Only just for in case then peer missed some lwwref update transactions because of network hiccups.

How to delete block/tree from storage

Delete single block:

hbs2 del <HASH>

Delete blocks recursively

hbs2 del -r <HASH>

Fetch block/tree

In background:

hbs2-peer fetch <HASH>

Wait tree/block to fetch

hbs2-peer -p <HASH>

May loop forever, if block/tree or their parts are not accessible to download

Store tree with metadata

hbs2-cli hbs2:tree:metadata:file [kw] ./yehatrmx.mp3

8vKn6y2EPonUZU8PKviWJB8gRxP6wnTYvFtwDbVLB5ja


hbs2-cli hbs2:tree:metadata:get 8vKn6y2EPonUZU8PKviWJB8gRxP6wnTYvFtwDbVLB5ja
((mime-type: "audio/mpeg; charset=binary") (file-name: "yehatrmx.mp3"))

curl -i http://localhost:5000/tree/8vKn6y2EPonUZU8PKviWJB8gRxP6wnTYvFtwDbVLB5ja
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Sat, 08 Feb 2025 09:29:03 GMT
Server: Warp/3.3.31
ETag: 8vKn6y2EPonUZU8PKviWJB8gRxP6wnTYvFtwDbVLB5ja
content-type: audio/mpeg; charset=binary
content-disposition: attachment; filename="yehatrmx.mp3"
Cache-Control: public, max-age=31536000, immutable

hbs2-cli hbs2:tree:metadata:file – calling a function from hbs2-cli.

This function accepts a dictionary of arguments and a file name.

Function kw is just a syntax sugar for making a dictionary from list:

hbs2-cli [kw a 1 b 2 c 3]
((a 1) (b 2) (c 3))

hbs2-cli [assoc c [kw a 1 b 2 c 3]]
(c 3)

All arguments will be stored in metadata section:

hbs2-cli hbs2:tree:metadata:file [kw note '"Yehat theme from SC2"'] ./yehatrmx.mp3
EXiiTEhcBHUj7oHYsMaFnaePCTNQhPziVJvtwhBGB9Y1

hbs2-cli hbs2:tree:metadata:get EXiiTEhcBHUj7oHYsMaFnaePCTNQhPziVJvtwhBGB9Y1
((mime-type: "audio/mpeg; charset=binary")
 (file-name: "yehatrmx.mp3")
 (note: "Yehat theme from SC2"))

Some of arguments may have a special meaning, like gk which points to hash of group key to encrypt the tree.

Yes, trees may be encrypted:

hbs2-cli hbs2:tree:metadata:file \
  [kw note '"Yehat theme from SC2"' gk 5vX2zxXYj4vRJqRnKzmSFcHKXwteyAiHjf5aScofVsAb] \
  ./yehatrmx.mp3

9txBjfr5wUKDowm2LoN9ZeoeiZ1Sh5G1T1EekuSBMn9j

What is this about: we’re just stored a tree encrypted with group key 5vX2zxXYj4vRJqRnKzmSFcHKXwteyAiHjf5aScofVsAb

Only members of this group key may read this tree now. For example:

hbs2-cli hbs2:groupkey:dump 5vX2zxXYj4vRJqRnKzmSFcHKXwteyAiHjf5aScofVsAb
; fancy group key
group-key-id Gs9swVTnuGAmZUqmN7gVu2dbGVDr8G4tbgRCFJbvYAtF
group-key-id-scheme basic1
group-key-timestamp 1738318754

member "AFFRDDokP4TzDv5gMGAEgSFVDR59NQbgPqxWN5iqcmhi"
member "GGferZDvxSYL4VTmHuiEebDnaJGHu9mTNznhLYoLKkqZ"
member "D4UQg95ndnnQYFG3tDmYJr6AGpKk1xbhvLidxDDugjbZ"

Member is a public key. The secret of the group key is encrypted by each of thouse public keys, so owners of the private key may decrypt the secret and decrypt the data, in this case – content of this particular tree.

I can decrypt the file while my key storage is mounted:

hbs2 cat 9txBjfr5wUKDowm2LoN9ZeoeiZ1Sh5G1T1EekuSBMn9j | hbs2 hash
Dtt3PKFuvyJtzWa46Bv8jP7atCqX3tuiqo6v3xf81irr

Now I unmount the key storage and try to do this again:

hbs2 cat 9txBjfr5wUKDowm2LoN9ZeoeiZ1Sh5G1T1EekuSBMn9j | hbs2 hash
GroupKeyNotFound 2
xyw95Bsby3s4mt6f4FmFDnFVpQBAeJxBFNGzu2cX4dM

The encrypted merkle three contains not encrypted system metadata (but user metadata is encrypted). I.e this tree may be downloaded and seeded by anyone, while only members may decrypt it’s data.