Serve Protomaps Tiles with NodeJS
Protomaps (PMTiles) is a relatively new storage format for storing and accessing vector tile data. The format is similar to MBTiles in that a single file can contain an entire tileset (of the entire planet if need be), though what differs is the retrieval method of individual tiles.
PMTiles are indexed in such a way that the data for a specific z/x/y tile can be accessed directly from the file using a specific byte range.
What this means in practice, is that PMTiles are inherently serverless. If you can access the file, you can load the data for individual tiles and render a map. The PMTiles javascript library has support for rendering maps client-side directly from a publicly accessible file, such as AWS S3.
See how a PMTiles setup compares to MBTiles. Notice no additional servers are needed.
So why use a server with PMTiles?
There are some scenarios when building an application you might want more control over: logging, authentication, bandwidth restrictions, or just more control over which data is served.
The PMTiles javascript library is primarily designed for use in a client environment, so this article will detail how to leverage the same library for accessing PMTile data in NodeJS.
Creating PMTiles
The creators of PMTiles made it very easy to get started. The pmtiles CLI can convert any existing MBtiles file to PMTiles with a single command:
pmtiles convert <input.mbtiles> <output.pmtiles>The Planetiler project which builds map tiles using the OpenMapTiles schema can also output directly to the PMTiles format. For this example, we’ll create a tileset for New York City.
# download JAR file
wget https://github.com/onthegomap/planetiler/releases/latest/download/planetiler.jar
# download data for New York state, limited to NYC with --bounds argument
# output to "nyc.pmtiles"
java -Xmx4g -jar planetiler.jar \
--download \
--area=new-york \
--bounds=-74.2546,40.4944,-73.6066,40.9452 \
--output=nyc.pmtilesFrom there, we should have a local nyc.pmtiles file that we can use in the next section.
Serving tiles from a local file
Let’s start with the most basic case. We have a PMTiles file, and we want to create an endpoint for serving this data to a client. We’ll assume the file is co-located on the same filesystem as the server. This means we can use Node’s built-in fs library for accessing the file. We'll use an architecture similar to the following.
The PMTiles library already has support for loading from with this FileAPISource class, but you’ll notice it’s based on the browser’s File API. We’re basically going to recreate this file, but swap out the browser API with the Node version.
To do that, we’ll create a new class FileSource that implements the pmtiles Source interface. In order for this to work, we’ll need to re-implement the getBytesfunction.
For any given tile, the PMTiles library will figure out the correct byte range to locate the data for that file, and it will pass in the offset and length for the range. It’s then up to us to implement the logic for retrieving those bytes from the file.
The fs.read function will read bytes from a file into a buffer, and supports specifying the range with the length and position arguments.class FileSource implements Source {
filename: string;
fileDescriptor: number;
constructor(filename: string) {
this.filename = filename;
this.fileDescriptor = fs.openSync(filename, 'r');
}
getKey() {
return this.filename;
}
// helper async function to read in bytes from file
readBytesIntoBuffer = async (buffer: Buffer, offset: number) => new Promise<void>((resolve, reject) => {
fs.read(this.fileDescriptor, buffer, 0, buffer.length, offset, (err) => {
if (err) {
return reject(err);
}
resolve();
});
});
getBytes = async (offset: number, length: number) => {
// create buffer and read in byes from file
const buffer = Buffer.alloc(length);
await this.readBytesIntoBuffer(buffer, offset);
const data = buffer.buffer.slice(
buffer.byteOffset,
buffer.byteOffset + buffer.byteLength,
);
return { data } as RangeResponse;
}
}The class can now be instantiated with our file to create a new PMTiles instance.
const source = new FileSource('./nyc.pmtiles');
const pmtiles = new PMTiles(source);And in our server, we can add a route to fetch a specific tile from the pmtiles file.
// handle tile request
app.get('/tile/:z/:x/:y.:format', async (req, res) => {
const { z, x, y, format } = req.params;
// load data based on tile request
const tile = await pmtiles.getZxy(Number(z), Number(x), Number(y));
if (!tile) {
res.send(404);
return;
}
const data = Buffer.from(tile.data);
// determine content-type header based on data
// (assume pbf for now)
const header = await pmtiles.getHeader();
switch (header.tileType) {
case 0:
console.log('Unknown tile type.');
break;
case 1:
res.header('Content-Type', 'application/x-protobuf');
break;
}
res.status(200).send(data);
});Putting it all together, I've added a simple HTML index page along with a MapLibre style object to be able to render the data in a map.
Fonts and glyphs are omitted for simplicity, so no text is visible.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
import fs from 'fs';
import path from 'path';
import { PMTiles, Source, RangeResponse } from 'pmtiles';
import express from 'express';
const app = express();
class FileSource implements Source {
filename: string;
fileDescriptor: number;
constructor(filename: string) {
this.filename = filename;
this.fileDescriptor = fs.openSync(filename, 'r');
}
getKey() {
return this.filename;
}
// async helper for reading file data into buffer
readBytesIntoBuffer = async (buffer: Buffer, offset: number) => new Promise<void>((resolve, reject) => {
fs.read(this.fileDescriptor, buffer, 0, buffer.length, offset, (err) => {
if (err) {
return reject(err);
}
resolve();
});
});
getBytes = async (offset: number, length: number) => {
// create buffer and read in byes from file
const buffer = Buffer.alloc(length);
await this.readBytesIntoBuffer(buffer, offset);
const data = buffer.buffer.slice(
buffer.byteOffset,
buffer.byteOffset + buffer.byteLength,
);
return { data } as RangeResponse;
}
}
// create new PMTiles instance from local file
const source = new FileSource('./nyc.pmtiles');
const pmtiles = new PMTiles(source);
const init = async () => {
app.get('/', (req, res) => {
res.sendFile(path.join(__dirname, 'index.html'));
});
app.get('/style.json', (req, res) => {
res.sendFile(path.join(__dirname, 'style.json'));
});
// serve TileJSON spec
app.get('/tilejson.json', async (req, res) => {
const metadata = await pmtiles.getMetadata();
const tileJSON = Object.assign({}, metadata);
tileJSON.tiles = [
"http://localhost:8080/styles/default/{z}/{x}/{y}.pbf"
];
tileJSON.minzoom = 0;
tileJSON.maxzoom = 14;
res.json(tileJSON);
});
// handle tile request
app.get('/styles/default/:z/:x/:y.:format', async (req, res) => {
const { z, x, y, format } = req.params;
// load data based on tile request
const tile = await pmtiles.getZxy(Number(z), Number(x), Number(y));
if (!tile) {
res.send(404);
return;
}
const data = Buffer.from(tile.data);
// determine content-type header based on data
// (assume pbf for now)
const header = await pmtiles.getHeader(); // this is cached?
switch (header.tileType) {
case 0:
console.log('Unknown tile type.');
break;
case 1:
res.header('Content-Type', 'application/x-protobuf');
break;
}
res.status(200).send(data);
});
console.log("Server running on port 8080");
app.listen(8080);
};
init();
Serving tiles from S3
While serving the tiles from a local file lets us use vector tiles without an additional dependency (SQLite) we still need to have the file co-located on the server that is running. One of the nice things about using byte-ranges, is we can serve the same data from a remote file, assuming the file protocol we’re using supports byte-range requests. To do this, we'll be setting up something like the following.
For this example, we’ll host the file on S3. To do this, we’ll create a new class (called S3Source) that will take the S3 file location (bucket & path) instead of the local path. We’ll also modify the getBytes function to fetch the byte range from S3, and convert the data to a binary buffer.
// helper function to convert a Buffer to ArrayBuffer
const bufferToArrayBuffer = (buffer: Buffer): ArrayBuffer => {
const arrayBuffer = new ArrayBuffer(buffer.length);
const data = new Uint8Array(arrayBuffer);
for (let i = 0; i < buffer.length; ++i) {
data[i] = buffer[i];
}
return arrayBuffer;
};
class S3Source implements Source {
bucket: string;
key: string;
constructor(bucket: string, key: string) {
this.bucket = bucket;
this.key = key;
}
getKey() {
return this.key;
}
getBytes = async (offset: number, length: number): Promise<RangeResponse> => {
const s3Params = {
Bucket: this.bucket,
Key: this.key,
Range: `bytes=${offset}-${offset + length - 1}`,
};
const s3Res = await S3.getObject(s3Params).promise();
// convert data from Buffer to ArrayBuffer of binary data
const data = bufferToArrayBuffer(s3Res.Body as Buffer);
return {
data,
etag: s3Res.ETag,
expires: s3Res.Expires?.toISOString(),
cacheControl: s3Res.CacheControl,
};
}
}1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
import path from 'path';
import { PMTiles, Source, RangeResponse } from 'pmtiles';
import AWS from 'aws-sdk';
import express from 'express';
const app = express();
AWS.config.update({
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
});
const S3 = new AWS.S3();
// convert a Buffer to ArrayBuffer
const bufferToArrayBuffer = (buffer: Buffer): ArrayBuffer => {
const arrayBuffer = new ArrayBuffer(buffer.length);
const data = new Uint8Array(arrayBuffer);
for (let i = 0; i < buffer.length; ++i) {
data[i] = buffer[i];
}
return arrayBuffer;
};
class S3Source implements Source {
bucket: string;
key: string;
constructor(bucket: string, key: string) {
this.bucket = bucket;
this.key = key;
}
getKey() {
return this.key;
}
getBytes = async (offset: number, length: number): Promise<RangeResponse> => {
const s3Params = {
Bucket: this.bucket,
Key: this.key,
Range: `bytes=${offset}-${offset + length - 1}`,
};
const s3Res = await S3.getObject(s3Params).promise();
// convert data from Buffer to ArrayBuffer of binary data
const data = bufferToArrayBuffer(s3Res.Body as Buffer);
return {
data,
etag: s3Res.ETag,
expires: s3Res.Expires?.toISOString(),
cacheControl: s3Res.CacheControl,
};
}
}
const source = new S3Source('com.ckochis.data', 'tiles/nyc.pmtiles');
const pmtiles = new PMTiles(source);
const init = async () => {
app.get('/', (req, res) => {
res.sendFile(path.join(__dirname, 'index.html'));
});
app.get('/style.json', (req, res) => {
res.sendFile(path.join(__dirname, 'style.json'));
});
app.get('/tilejson.json', async (req, res) => {
const metadata = await pmtiles.getMetadata();
const tileJSON = Object.assign({}, metadata);
tileJSON.tiles = [
"http://localhost:8080/styles/default/{z}/{x}/{y}.pbf"
];
tileJSON.minzoom = 0;
tileJSON.maxzoom = 14;
res.json(tileJSON);
});
// handle tile request
app.get('/styles/default/:z/:x/:y.:format', async (req, res) => {
const { z, x, y, format } = req.params;
// load data based on tile request
const tile = await pmtiles.getZxy(Number(z), Number(x), Number(y));
if (!tile) {
res.send(404);
return;
}
const data = Buffer.from(tile.data);
// determine content-type header based on data
// (assume pbf for now)
const header = await pmtiles.getHeader();
switch (header.tileType) {
case 0:
console.log('Unknown tile type.');
break;
case 1:
res.header('Content-Type', 'application/x-protobuf');
break;
}
res.status(200).send(data);
});
console.log("Server running on port 8080");
app.listen(8080);
};
init();
