Stream real-time data with PubNub in Node.js and Python

You build the stream in three parts: a Node.js app that publishes Wikimedia's changes with the PubNub SDK, the PubNub data streaming demo to watch them in the browser, and a Python script that publishes with HTTP pipelining.

Part 1. Publish Wikimedia changes with the SDK

Wikimedia publishes every change to its wikis as EventStreams, using server-sent events. This app reads the recent changes stream. To see the raw feed first, run:

curl -N https://stream.wikimedia.org/v2/stream/recentchange

Each data: line is one change, as JSON. Press Ctrl+C to stop.

Open real-time-pubnub-sdk/index.js and replace YOUR_PUBLISH_KEY and YOUR_SUBSCRIBE_KEY with your keys. The app reads the stream with the eventsource package and creates a PubNub client:

Every client needs a User ID. A server app like this one can use a fixed one. For each change, the app prints one line and publishes the whole event to the pubnub-wikipedia channel. If a publish fails, for example because a key is wrong, it prints the reason:

Install the dependencies and start the app. package.json pins the versions it was tested with: pubnub 13.1.3 and eventsource 5.1.3.

cd real-time-pubnub-sdk
npm install
npm start

A line appears for every change, from every Wikimedia wiki:

commons.wikimedia.org categorize Category:Images from Mapillary uploaded with Curator
en.wikipedia.org edit Hung Up
www.wikidata.org edit Q141896144

To check that the messages reach PubNub, open the Admin Portal, select Build → Debug Console, choose your app and keyset, and subscribe to pubnub-wikipedia. Leave the app running for part 2.

The SDK has no hard publish rate limit. The API limits page recommends 10 to 15 messages per second per channel, because a slow subscriber can fall behind its connection's message buffer. Wikimedia's stream is often faster than that; filter it to one wiki, as the comment in the code shows, if a subscriber can't keep up.

Part 2. Watch the stream in the data streaming demo

The PubNub data streaming demo subscribes to several live streams, one of them Wikipedia changes. Point it at your keyset to see your own stream. Clone it in a new terminal:

git clone https://github.com/PubNubDevelopers/Real-Time-Data-Streaming-Demo.git
cd Real-Time-Data-Streaming-Demo

In js/streams.js, find the entry with id: 'wikipedia'. Set its subKey to your subscribe key and keep its channel on pubnub-wikipedia, the channel the app publishes to.

The demo's own Wikipedia stream sends a different message format, so in js/index.js, replace the addFormattedMessageWikipedia function with the one in demo-changes/addFormattedMessageWikipedia.js of the tutorial repository:

It adds the page titles and user names as text, not HTML, because anyone can edit a wiki: a title with HTML in it shows as text instead of running in your page. payload.timetoken is PubNub's timestamp in 100-nanosecond units, so dividing it by 10,000 gives milliseconds for a JavaScript Date.

Serve the folder and open http://localhost:8000/:

python3 -m http.server 8000

Select Wikipedia Changes (Live) in the stream menu. With the Node.js app from part 1 still running, the changes appear as they're published:

If nothing appears, check that the Node.js app is running and prints no Publish failed lines, that both use the same subscribe key, and that the channel is pubnub-wikipedia in both. Stop the app with Ctrl+C when you're done.

Part 3. Publish with HTTP pipelining and watch with HTTP streaming

The SDK sends each publish as its own HTTP request. For very high throughput, you can skip the SDK and pipeline instead: write many publish requests to one connection without waiting for replies, then read the replies as they come back. The script in real-time-pubnub-stream/pubnub-stream3.py is a Python 3 port of an HTTP pipelining gist from PubNub. HTTP pipelining and the streaming URL below aren't part of the documented REST API, so talk to PubNub before you build on them in production.

Open the script and replace YOUR_PUBLISH_KEY and YOUR_SUBSCRIBE_KEY with your keys. It connects to ps.pndsn.com, the host the PubNub SDKs use, over TLS:

Every 10 seconds, it starts a round: it opens a connection and sends a random number of publish requests, a few thousand to about 20,000. Each message is a random UUID, as URL-encoded JSON. The requests go out BATCH at a time: the script writes 1,000 requests in one go without waiting, then reads their replies and counts the ones PubNub accepted. One thread does both, because a TLS socket can't be read and written from two threads at once. If PubNub rejects a batch, for example because a key is wrong, the script prints the reason and ends the round:

Each request is a plain HTTP GET to the same publish endpoint the SDK calls:

Before you run it, open the stream of the streamtest channel in a browser tab, with your subscribe key in the URL:

https://ps.pndsn.com/stream/YOUR_SUBSCRIBE_KEY/streamtest/0/10000

Then run the script from the repository folder:

python3 real-time-pubnub-stream/pubnub-stream3.py

After 10 seconds, it starts the first round and reports on it:

creating 8063 UUIDs
We will send a random quantity of messages every 10 seconds, each contains a randomly selected UUID.
Sending 4707 messages.
It took 6.843616008758545 seconds to send 4707 messages
PubNub accepted 4707 of 4707 messages

In our tests, one connection published 700 to 1,000 messages a second, all of them accepted. The browser tab fills with the UUIDs faster than you can read them. The browser doesn't keep the stream open forever, so reload the tab if it stops updating.

The script starts a new round every 10 seconds until you stop it with Ctrl+C. At this rate it uses a free account's 1 million transactions in about 20 minutes, so don't leave it running. Testing keysets are also rate-limited and not meant for sustained traffic like this.