From what I can tell based on the source code, the terrain is organized as an array of "zones". Each zone consists of a 128x128 grid of 12-bit height field entries (the .hgt file) and a 64x64 grid of 16-bit material entries (the .mat file). The material entries are a bit complicated since they pack a lot of data into a very small space.
From least-significant bit to most-significant bit:
2 bits at shift 0 (0x0003): Variant
2 bits at shift 4 (0x0030): Rotation
1 bit at shift 6 (0x0040): Flip
1 bit at shift 7 (0x0080): Cap
4 bits at shift 8 (0x0F00): Next
4 bits at shift 12 (0xF000): Base
If Base and Next are the same, it's a Solid tile
If Cap is 0, it's a CapTo tile where Base touches Next on only one side
If Cap is 1, it's a DiagonalTo tile where Base touches Next on two adjacent sides
Variant selects from several similar textures for the tile (_A0, _A1, _A2, and _A3)
Rotation, Flip, and Cap combine to select the texture coordinates for the tile:
0: (u, v) cap 0
1: (-v, u) cap 90
2: (-u, -v) cap 180
3: (v, -u) cap 270
4: (-u, v) flip across z
5: (-v, -u) flip across -x/z diagonal
6: (u, -v) flip across x
7: (v, u) flip across x/z diagonal
8: (u, v) diag 0
9: (-v, u) diag 90
10: (-u, -v) diag 180
11: (v, -u) diag 270
12: (-v, -u) flip across -x/z diagonal
13: (u, -v) flip across x
14: (v, u) flip across x/z diagonal
15: (-u, v) flip across z
Fun, eh?
