IoT REBOOT – Apple HomeKit (Siri), Raspberry Pi, node-red, Homebridge

Simplifying the IoT setup

IoT-Reload

A new version of the IoT server

The goal of this project is to install a Internet of Things (IoT) server that can control “things” AND, more importantly, communicate with Apple’s Siri and HomeKit. Initially, we will install a server on a Raspberry Pi. This server uses node.js extensively, but instead of doing all the programming ourselves, we use existing code, with node-red and plugins as an interface. Node-red will handle communications between node.js programs, the hardware of our IoT things and Apple Siri (and HomeKit).

In my system, I use node-red to control (and read data from) my own IoT things. I then use node-red (node.js/json/some programming) to process that information. I also use MQTT to exchange information between devices (to/from IoT things and between them). Then I use Homebridge to send and receive information with HomeKit/Siri. I then use Home, the HomeKit program from apple, to control everything automatically.

It’s a lot of technology, but actually very little setup and programming. I use pre-made utilities (thanks to every one who develops all this!).

In two previous posts (here and here), I explained how to setup a simple server, running on a Raspberry Pi (2 or 3), that would serve as a bridge to Apple Siri to control HomeKit accessories. Since the posts were published, many contributors have worked hard to integrate software components on various platforms. Raspberry Pi is evolving through its software packages. Apple is even talking about opening up HomeKit (a bit). This leads to things like Ikea Tradfri connecting to HomeKit.

This post is about a new installation procedure for installing node-red, Homebridge (HAP-Nodejs) and MQTT on a Raspberry Pi. I tested the procedure and was able to get a working HomeKit Server in about an hour! Of course, if you read this article a year (a month?) after I published it, the procedure might be even simpler! I will update this post if I find a better procedure.

Configuring the Raspberry Pi as our IoT server

I use a Raspberry Pi 2 as my home server. The Pi 3 can be used interchangeably and the Pi Zero should work too (not tested). I just happened to have a 2 in stock and I don’t mind having the server linked to my network with a cable instead of Wifi.

The simplest procedure to initially configure the Pi is described on raspberry’s web site. Basically, just download Raspbian and install using Etcher. I’m using Raspbian Jessie, from that link. Please download the full version, with the graphical interface.

A couple of tricks

Although you can setup your server using the graphical interface on your Raspberry Pi, most of the procedure described here is done via the command line, in Linux, running on the Raspi. If you are uncomfortable with the command line, you probably should not work on this project anyway. Spend some time working in command line mode and come back!

I prefer to configure and maintain my servers using the command line only, and I also prefer to do this remotely. This allows me to have servers and remote stations anywhere on the network, without having to connect them to a screen or keyboard. Just after you burn the operating system on the micro-SD card, and before you put it in your Pi, I suggest you slightly modify the “Boot” partition. All you need to do is take the SD card out of your computer and re-insert it. The computer should show you a “Boot” disk on Windows or MacOS.

Newer version of Jessie disable ssh by default. To activate it, add a file called “ssh” to the Boot disk/partition. Just en empty file. In Linux/MacOS: “touch ssh” will do it. This will activate ssh on the first boot of your Raspi. The file will be deleted automatically after that. This way, you can do all the configuration from the command line, remotely.

You can also modify a file and boot with Wifi already activated. Read the answer in this link.

After the initial boot, you will need to get the IP address of your new Raspi. This will vary greatly depending on your network configuration, but in general, your router should show you a new device on the network with its address. If you can’t find it, boot you Pi attached to a screen (use the HDMI connector) and find its IP address by opening a terminal console and typing

ifconfig -a

From now on, you can access your Pi using Putty on Windows, Terminal on a Mac, and the command line in Linux. I even use an iPad sometimes. You might want to disable the graphical interface (normal boot mode). But you will want to keep the graphical interface setup on the Raspi. If you use a graphical dashboard, running on the Pi, you will want to boot in graphical mode.

ssh pi@192.168.2.xxx     <=replace with IP address of your Pi!

It’s now time to adjust you Pi to your time zone and change its network name and change the default password. Use

sudo raspi-config

And now we install some software!

Installing IoT Software

Were are going to use node-red, Mosquitto/MQTT and homebridge on the server.

But first, learn to use

sudo apt-get update

sudo apt-get upgrade

and use these two commands every time you install new software on your Raspberry Pi. The first one update the software catalogs (repositories) and the second one upgrades installed packages, as needed. This will keep you Pi up to date.

Node-red

Node-red is the visual programming interface that I use.

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways.

It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.

As of this writing, Raspbian Jessie comes with node-red pre-installed, but the installed version of node.js is an older one. Node.js is necessary to run node-red and the rest of our IoT stuff. The old version of this post made you install node.js “by hand”, but it’s now all included in a nifty process already present on your Pi:

update-nodejs-and-nodered

This is a marvellous script! Read the details here.

If the previous command doesn’t work, try: (read about it here)

bash <(curl -sL https://raw.githubusercontent.com/node-red/raspbian-deb-package/master/resources/update-nodejs-and-nodered)

All we really need now is to have node-red start when the Pi boots. This is very simple on a recent version of Jessie:

sudo systemctl enable nodered.service

Reboot to activate the service (and check for errors, while you’re there).

sudo reboot

If you want to play with node-red now, please go ahead. There are many tutorials available. One in particular, from IBM, explains the whole setup of an intelligent doorbell. Read it to get an idea of what the rest of the project is going to be. My install procedure is simpler, but the explanations of both hardware and software usage are excellent.

Mosquitto

MQTT, which originally was an acronym for Message Queue Telemetry Transport, is a lightweight message queue protocol designed for small data packets sent across high latency, low bandwidth links. MQTT is a fairly simple protocol and it’s perfect for Internet of Things projects.

MQTT will allow us to exchange information, using a simple protocol, between the component of our IoT system. It’s as simple as sending a message like “light1/On” or “light1/Off”. MQTT takes care of the synchronisation, message storage, etc. The syntax is really simple. Mosquitto is the “most better version” of an MQTT server for the Raspberry Pi.

Mosquitto is now available on the main Raspbian repository, so installation is very simple:

sudo apt-get install mosquitto mosquitto-clients

Mosquito will install its service and will be there at reboot.

Homebridge

Homebridge is a lightweight NodeJS server you can run on your home network that emulates the iOS HomeKit API. It supports Plugins, which are community-contributed modules that provide a basic bridge from HomeKit to various 3rd-party APIs provided by manufacturers of “smart home” devices.

Since Siri supports devices added through HomeKit, this means that with Homebridge you can ask Siri to control devices that don’t have any support for HomeKit at all.

Homebridge is quite invisible on the server. IT runs in the background and is seen by Apple’s HomeKit application Home as a standard bridge. So when we create new IoT accessories, they are made visible in the Home application exactly as if you had bought them! We will communicate with Homebridge using an MQTT plugin. This way, we can configure and communicate with it.

Install Homebridge

In the previous version of this procedure, I installed HAP-nodejs separately. This is now included in Homebridge installation. The Homebridge installation is well detailed by its creator here (and here for a semi-manual install on a Raspi). Basically, it’s just:

sudo apt-get install libavahi-compat-libdnssd-dev

sudo npm install -g --unsafe-perm homebridge

The  “npm” command (I think it’s an abbreviation for “node program manager”) is used to install node.js modules. Homebridge installation will take care of the dependancies (like HAP-nodejs). The -g parameter is important. This will install Homebridge globally. This is important to be able to get it going on boot.

Make Homebridge execute at boot

Since we also want Homebridge to start at boot, we need to do a few more things. The details are here. Please read them. It looks complicated but it’s just two files and a directory that need to be created:

Download the two files at the top of the page and place homebridge under /etc/default and homebridge.service under /etc/systemd/system on your Raspberry Pi. This is how a service is created on newer versions of Jessie.

You can also copy and paste the code:

  • sudo nano /etc/default/homebridge

and copy this code into the file

# Defaults / Configuration options for homebridge
# The following settings tells homebridge where to find the config.json file and where to persist the data (i.e. pairing and others)
HOMEBRIDGE_OPTS=-U /var/lib/homebridge

# If you uncomment the following line, homebridge will log more 
# You can display this via systemd's journalctl: journalctl -f -u homebridge
# DEBUG=*

Then do

  • sudo nano /etc/systemd/system/homebridge.service

and copy this code into the file

[Unit]
Description=Node.js HomeKit Server 
After=syslog.target network-online.target

[Service]
Type=simple
User=homebridge
EnvironmentFile=/etc/default/homebridge
# Adapt this to your specific setup (could be /usr/bin/homebridge)
# See comments below for more information
ExecStart=/usr/local/bin/homebridge $HOMEBRIDGE_OPTS
Restart=on-failure
RestartSec=10
KillMode=process

[Install]
WantedBy=multi-user.target

In order to use the systemd service as is, the following folders and user have to exists:

  • A system user named “homebridge". You can easily create this user with
    sudo useradd -M --system homebridge
  • A directory called /var/lib/homebridge, writable by the user created above, and a corresponding config.json file in that directory. Homebridge by default looks for its configuration in /home/<username>/.homebridge. This is unsuitable for services and the -U /var/lib/homebridge flag ensures the config is read from a different place.
sudo mkdir /var/lib/homebridge

sudo chmod 755 /var/lib/homebridge

Then Enable and run the service (first time) with the following commands:

systemctl daemon-reload
systemctl enable homebridge
systemctl start homebridge

You can check the status of the service by calling

systemctl status homebridge

On subsequent reboots, it should start automatically, if not, use the journalctl -u homebridge to check the error cause, or look into /var/log/syslog.

Our Homebridge installation needs a bit more configuration.

Configuring Homebridge

cd /home/pi

This is the main directory for user Pi. We will configure Homebridge here and then copy the file to the proper directory.

cd .homebridge

This is a hidden directory. Because of the “.” in front of the name. The directory will probably be empty unless you tried starting Homebridge. We will create a basic configuration file:

sudo nano config.json

and the copy these json lines into the file:

{
 "bridge": {
    "name": "Homebridge",
    "username": "CC:22:3D:E3:CE:30",
    "port": 51826,
    "pin": "031-45-154"
 },

"description": "This is the MQTT connector",
 "platforms": [
 {
    "platform": "mqtt",
    "name": "mqtt",
    "url": "mqtt://192.168.2.xxx”, <= address here
    "topic_prefix": "homebridge",
    "username": "foo",
    "password": "bar"
 }]
 }

If you were to use Homebridge by itself, without node-red, you would need to reference your accessories in the configuration file. But we will be doing all this in node-red and it will take care of the interface for us. Also, since we made Homebridge bootable as a service, we need to copy this configuration file in the directory we created earlier:

sudo cp /home/pi/.homebridge/config.json /var/lib/homebridge

and then make the file available to the new user we created:

sudo chown homebridge:homebridge /var/lib/config/config.json

sudo chmod 755 /var/lib/homebridge/config.json

Finally, we need to install 3 plugins for our server. The first 2 are important and the third one is optional, but we will use it later as an example.

Installing IoT plugins

Installing plugins for node is done using “npm”. Note that node-red plugins are node modules, so are installed the same way, except for the -g parameter.

First, we will install a plugin to let MQTT communicate with homebridge:

sudo npm install -g homebridge-mqtt

then a plugin to install a nice dashboard in node-red:

cd /home/pi/.nodered

Notice the “.” in front of the directory name. This is another “hidden” directory. Make sure that you are installing this plugin in that directory. Actually it will be installed in a sub-directory called “node_modules”. But installing it here will make this plugin available to node-red, and will create the proper visible functions. In the future, when you install a node-red plugin, come back to this directory.

npm install node-red-dashboard

Then we install a small plugin to play with our completed installation:

npm install node-red-node-weather-underground

We’re done!

The next post will contain examples.

Posted in Electronics, IoT | 3 Comments

Apple HomeKit on iOS with Arduino (ESP8266) through Raspberry Pi – Part 2

Apple HomeKit on iOS with Arduino (ESP8266) through Raspberry Pi

Note: This post, and the previous one (part 1), are obsolete. They have been replaced by this one, offering a simpler method to get started. They are left here for reference.

Part 1 of this article explained how to install a HomeKit server on a Raspberry Pi. This will allow you to add new, DIY accessories that can receive commands from Siri.

Part 2 (this article) explains how to install an MQTT server that can be a “bridge” between Siri (through you HomeKit server) and you hardware.

IoT Part 2

MQTT is a communication protocol that will let you send and receive simple messages from one computer to another. You need one server a a number of clients. We will setup the server on our Raspberry Pi, but there are servers available publicly.

Install the MQTT server: Mosquitto

There are other MQTT servers available but this one works well and is easy to install.

sudo wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key
cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-wheezy.list
sudo apt-get update
sudo apt-get install mosquitto

This will add the mosquitto repo to your list, then the last instruction actually uses the standard apt-get install.

Next,  install the Mosquitto clients to test your server (if you want). They are the actual commands to use your Raspberry Pi as an MQTT client.

sudo apt-get install mosquitto-clients

This link has many good pointers to configure the Mosquitto server to start and stop as a service, but this simple installation will be ok for testing.

Let’s test our mosquitto server. Open a new terminal window on your Raspberry Pi (or an SSH session if you are accesing it remotely) and type

mosquitto_sub -d -t hello/world

This will “subscribe” to the “hello” topic (-t), “world” sub-topic on your server (localhost). This can be ANYTHING you want. There is no real format and the / or /’s are optional.  The terminal will be executing MQTT in “subscribe” (think if it as “receive” or “listen”) mode and will just wait for the proper incoming message, meaning with the proper topic.

Now we need to publish a message (-m) on the same topic (-t), but in a new terminal window (or SSH session):

mosquitto_pub -d -t hello/world -m "Hello from Terminal window 2!"

The first window should show the message

Client mosqsub/12234-raspberry received PUBLISH (d0, q0, r0, m0, 'hello/world', ... (29 bytes))
Hello from Terminal window 2!

In my environment, I have one MQTT server on a Raspberry Pi and messages coming and going from other Raspberry Pis and sensor equipped Arduinos.

A particular device can send or receive MQTT messages and can also do both at the same time.

EPS8266 Fifi Micro-controller

For this article, I’m using an Arduino type of micro-controller that has a Wifi stack. Someone has written an Arduino library that handles MQTT communications, and being Wifi, this Arduino also has a Wifi library taking care of communications.

I’m using an Adafruit Huzzah ESP8266 micro controller. It has Wifi on board and is compatible with the Arduino environment. It’s not strictly an arduino, and the processor is not the typical Atmel 328, but Adafruit has written everything you need to program it from the Arduino IDE. Any wifi-enabled Arduino could be used, but the the ESP8266 is dirt cheap (10$) and is very compact. Please use the link above to learn how to dowload the appropriate Arduino library and run some tests for Wifi connections.

Now, let’s test the ESP8266 with a simple sketch that will send a message on the topic “/hello”. The message is “world” and is sent every minute. The sketch will also listen to the topic “hello/world” and print incoming messages. So if everything goes well, it will print the message it just sent to the MQTT server!

/*
  Basic ESP8266 MQTT example

  This sketch demonstrates the capabilities of the pubsub library in combination
  with the ESP8266 board/library.

  It connects to an MQTT server then:
  - publishes "hello world" to the topic "outTopic" every two seconds
  - subscribes to the topic "inTopic", printing out any messages
    it receives. NB - it assumes the received payloads are strings not binary
  - If the first character of the topic "inTopic" is an 1, switch ON the ESP Led,
    else switch it off

  It will reconnect to the server if the connection is lost using a blocking
  reconnect function. See the 'mqtt_reconnect_nonblocking' example for how to
  achieve the same result without blocking the main loop.

  To install the ESP8266 board, (using Arduino 1.6.4+):
  - Add the following 3rd party board manager under "File -> Preferences -> Additional Boards Manager URLs":
       http://arduino.esp8266.com/stable/package_esp8266com_index.json
  - Open the "Tools -> Board -> Board Manager" and click install for the ESP8266"
  - Select your ESP8266 in "Tools -> Board"

*/

#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.

const char* ssid = "default";
const char* password = "leborddulac";
const char* mqtt_server = "192.168.2.180";
const char* host = "hello/world"; //used to identify topic. This is the string of
//  characcters used in the *_accessory.js file to identify the topic

WiFiClient espClient;
PubSubClient client(espClient);
long lastMsg = 0;
char msg[50];
int value = 0;

void setup() {
  pinMode(BUILTIN_LED, OUTPUT);     // Initialize the BUILTIN_LED pin as an output
  digitalWrite(BUILTIN_LED, HIGH); // ...and turn if off (reversed logic for the esp8266)
  Serial.begin(115200);
  setup_wifi();
  client.setServer(mqtt_server, 1883); //connect to the mqtt server at address above
  client.setCallback(callback);
}

void setup_wifi() {

  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void callback(char* topic, byte* payload, unsigned int length) {
 Serial.print("Message arrived [");
 Serial.print(topic);
 Serial.print("] ");
 String mypayload = " ";
 for (int i = 0; i < length; i++) {
 Serial.print((char)payload[i]);
 mypayload[i] = (char)payload[i];
 }
 Serial.print(" Payload: ");
 Serial.print(mypayload);
 Serial.println();
 String mytopic = String(topic);
 if (mytopic == host) { // Is this the right Topic?
 // Switch on the LED if the payload in "on"
 //if ((char)payload[1] == 'n') { if (mypayload == statusOn) {
 if (mypayload == statusOn) {
 digitalWrite(BUILTIN_LED, LOW); // Turn the LED on (Note that LOW is the voltage level
 // but actually the LED is on; this is because
 // it is acive low on the ESP-01)
 //client.publish(host, "Powered");
 } else {
 digitalWrite(BUILTIN_LED, HIGH); // Turn the LED off by making the voltage HIGH
 //client.publish(host, "NOTPowered");
 }
 }
}

void reconnect() {
 // Loop until we're reconnected
 while (!client.connected()) {
 Serial.print("Attempting MQTT connection...");
 // Attempt to connect
 if (client.connect("ESP8266Client")) {
 Serial.println("connected");
 // Once connected, publish an announcement...
 client.publish("outTopic", "hello world");
 // ... and resubscribe
 client.subscribe("#");
 } else {
 Serial.print("failed, rc=");
 Serial.print(client.state());
 Serial.println(" try again in 5 seconds");
 // Wait 5 seconds before retrying
 delay(5000);
 }
 }
}

void loop() {

 if (!client.connected()) {
 reconnect();
 }
 client.loop();

 long now = millis();
 if (now - lastMsg > 60000) {
 lastMsg = now;
 ++value;
 snprintf (msg, 75, "hello world #%ld", value);
 Serial.print("Publish message: ");
 Serial.println(msg);
 client.publish("outTopic", msg);
 }
}

Once you ESP8266 is communicating with your MQTT server, most of the hard work is done. You now have an Arduino type micro-controller linked to your MQTT network.

You could decide to go no further and not link this to HomeKit and Siri. In fact, most of my sensors use MQTT to communicate together on my Wifi network, and these sensors are connected to Arduino, Raspberry Pi and other computers. In a future article, I will talk about HomeKit in more details, and eventually I will explain my node-red setup and interface.

Posted in Arduino, Electronics | 7 Comments

Apple HomeKit on iOS with Arduino (ESP8266) through Raspberry Pi – Part 1

The Internet of Thinks (IoT), using and iPhone, Siri, HomeBridge, MQTT and an Arduino – Part 1

Note: This post, and the next one (part 2), are obsolete. They have been replaced by this one, offering a simpler method to get started. They are left here for reference.

The global project is shown in this drawing:

iot-image_part-1

Part 1 (this article) is in yellow. Part 2 (next article) will cover the Mosquitto server (just below) and the DIY HomeKit Device. The rest will be detailed in future articles, over the next few weeks. Continue reading

Posted in Uncategorized | 2 Comments

Review: Zivix PUC+ for Bluetooth Midi Wireless

I’ve been a fan off wireless MIDI for a while. I built a few prototypes and I still use some of the devices I made. I might have found a replacement: Zivix just came out (fall 2015) with a wireless Midi transmitter, called the puc+ (puc plus), that is using Bluetooth LE (BLE) to exchange Midi information between any Midi device and a computer (Mac) or iPhone/iPad. This review is for the puc+. Zivix made a previous version called “puc” a couple of years ago (after a successful Indiegogo campaign). It was using a Wifi dedicated network connection. The new device is definitely more useful.

(By the way, you might have found this page doing a google search on puc+, puc plus or puc. Product names with special characters (+) are not search engine friendly…)

 PUC+ top Continue reading

Posted in Electronics, Music equipment, Uncategorized | Tagged , , | 4 Comments

M4L: Max for live patch for better guitar solo

Max for Live using Pitch Bend

When using a keyboard to play guitar, you have to use Midi Pitch Bend (Pitchbend or PB) quite often to emulate the effect of bending a string on a real guitar. The problem is that PB affects the sound synthesizer, not the Midi notes. Modifying the synth sound means that every note currently playing will be bent. On a real guitar, it’s generally the highest note that is bent, and sometimes the second highest if you keep pushing on the string, far enough to hit the second string.

So I programmed a small Max for Live (M4L) patch for Ableton Live to let me bend one note only, even when playing chords.

If you use Live, this will mean something to you. Keep reading.

The device is made of two patches. The first is a little program that will identify the highest note and pass it to the second patch. The second patch receives notes sent by the first one and passes them to the synth.

To install, you have to place the first patch in front of your synth in you guitar (or other instrument) track.

Sending patch in front of synth

Sending patch in front of synth

Continue reading

Posted in Ableton, M4L, Max for Live | Tagged , | Leave a comment

VC-1: Volume control for guitar using Arduino

As explained here, here and here, the VC-1 is a volume control for guitar (or bass) that can use a potentiometer, a variable analog signal or a Midi command to change the output volume.

VC-1-closeup

Continue reading

Posted in Arduino, Electronics, Music equipment, VC-1 | 3 Comments

Akai MPK261: One more thing using Sysex…

In a previous post, I explained how to control preset changes on the mPK261 using SysEx commands. In this post, I explain how to get the MPK2 series to show some tricked pad colors.

Continue reading

Posted in MPK261, Music equipment | 14 Comments

MIDI Experiments: Arduino as master

Arduino as a Midi master

The previous two posts explained how standard MIDI can supply some current at 5 Volts if implemented according to the MIDI standards. In this post, I go a little further and explain how an Arduino, set up as a MIDI device, can power a second Arduino using a standard MIDI cable.

Continue reading

Posted in Arduino, Electronics | Leave a comment

MIDI experiments: Stealing power for an Arduino

In the previous post, I explained a simple circuit that will indicate if a MIDI OUT port can provide 5 Volts and a little bit of current between pin 2 (ground) and pin 4 (+5 Volts). Not all MIDI devices are wired that way. Some leave pin 2 disconnected (bad) while others connect it to chassis ground (often the same as circuit ground) and a few use a micro-controller pin as a false ground (!?).

What if 5 Volts is available?

Here’s an old Arduino Duemilanove hookup to to my Akai MPK261:

arduino-midi-power

If you have sharp eyes, you will see the power LED lit right beside the Arduino word. Continue reading

Posted in Arduino, Electronics | Leave a comment

MIDI experiments: A simple MIDI tester

While doing some research to build a simple MIDI tester, I started to study the “real world” MIDI electrical specifications out there. I also experimented with various MIDI equipment and measured some voltages.

A simple MIDI tester

MIDI specifications imply that a MIDI OUT circuit always provides +5 Volts, on pin 4, in reference to circuit ground, present on pin 2. Pin 5 carries the actual MIDI signal and switches from +5 volts to zero Volts, referenced to the ground present on pin 2.

To test if a MIDI OUT circuit adheres to the specs, I use this simple circuit:

simple-midi-tester-1 Continue reading

Posted in Electronics, Music equipment | Tagged | 2 Comments