Ready to Google your Smart Thing?
An Express.js train ride down the Actions on Google smart home trail
I remember when I got my first Google Home Mini. There was something viscerally satisfying about telling it to play a song for me on Spotify and being able to adjust the volume by just saying “Hey Google, turn the volume up!”. I’m that guy who constantly annoys any house guests by showing them all the different features like some type of 8 year old in the 90’s who wouldn’t shut up about his new Power Rangers toy. I know I’m dating myself, but you secretly know you liked them too… Anyway, I got kind of addicted to that feeling of my little electronic butler turning my voice into action. So much that I wanted to be able to teach my Google Assistant to do even more. If you feel the way I do, and I know you do, keep reading and join me on a crazy train ride to Google land.
Welcome to the Express.js train ride down the Actions on Google smart home trail! Google has a bunch of great tutorials for getting started making actions in their Codelabs, but many of them use Firebase and I really like developing and running my server software locally. So on this train ride we’re going to develop a smart home action for the Google Assistant using Express.js and ngrok.
Stop 1: Account Linking
Welcome to Stop 1: Account Linking. Here is where the Google Action project gets created and linked up to the Express.js server. To create the project, we start by going to https://developers.google.com/assistant and from there the Actions Console at https://console.actions.google.com. On the console, select “New project”.
I’ll name my Google Actions project “Google My Smart Thing”.
There are all different types of actions to choose from including conversational apps that use technologies such as Dialogflow. However, I want to use Google smart home, so I’ll just click the smart home option on the next page.
After that, we’ll land on the console page for our project with a Develop tab. I think “The Smart Thing Train” is a good name.
Alright! Now we can setup account linking from the Overview page. This option may not show up if you didn’t choose Smart Home for your development experience.
Under Account creation, I’ll select “No, I only want to allow account creation on my website” and then “Next”. Now I’ll choose OAuth with an authorization code for the Linking type. This train is getting started!
That OAuth Client needs some info. A client ID of “mySmartThing” and a secret of “shhhhh” will do the trick. Now what’s up with these Authorization and Token URLs?
I can’t really go any further until I get those darn OAuth URLs… I guess now we’ve got to bring in our own server. To get things a rollin’, I’ve already put a small Express.js server for you here on github: https://github.com/jcartw/google-your-smart-thing. Note that you’ll start out on the 1-account-linking branch. There are different branches for each stage of this train ride.
I’ll open up my favorite terminal and run a few commands.
$ git clone https://github.com/jcartw/google-your-smart-thing.git
$ cd google-your-smart-thing
$ yarn install
$ yarn startstarting server on port 9000
Now I’ve got a server running on 9000 on localhost. We’ll just use some good ole’ ngrok glue to link this up to the Google Action project. If you need to install ngrok, our old friend npm is always here to help.
$ npm install -g ngrokWe’ll use ngrok to provide a public URL that points to our local server on port 9000. We’ve even got an HTTPS connection. Your subdomain will be different, but I’ll be using https://d8a236bd.ngrok.io.
$ ngrok http 9000ngrok by @inconshreveable (Ctrl+C to quit)
Session Status online
Session Expires 7 hours, 59 minutes
Update update available (version 2.3.35, Ctrl-U to update)
Version 2.2.8
Region United States (us)
Web Interface http://127.0.0.1:4040
Forwarding http://d8a236bd.ngrok.io -> localhost:9000
Forwarding https://d8a236bd.ngrok.io -> localhost:9000
Connections ttl opn rt1 rt5 p50 p90
0 0 0.00 0.00 0.00 0.00
Now, let’s take a gander at our server code to see what REST routes are available to use for OAuth. These are all within server.js.
So for our OAuth flow we’ve got the following routes:
- GET /authorize
- POST /login
- POST /token
I’m only going to provide the /authorize and /token routes for our OAuth Client information page. Oh, and what about /login you ask? What about that little guy? We’ll worry about that little guy later…
I’m going to go with “Next” and skip the Configure your client step. The Testing instructions section is where we would provide actual username and password information for Google to test the action if we were to publicly launch. I’m just going to put “no username or password required” here and continue. We’re just about ready to roll down the tracks!
The Express.js server handles both OAuth and the Google Action fulfillment. I’m sure you noticed the POST /fulfill route in server.js. The location of that route also needs to be provided to Google. On the Overview page, I’m going to select “Add Action(s)” under “Build your Action”.
I’ll use my ngrok URL again here and provide the fulfillment address of the server. Save, and now you’re good as gold.
But what’s a smart home without a smart home app? Make sure to download Google Home on your iPhone or Android mobile device.
I’m going to go into my Google Home app and select “Add”, “Set up device”, then the icon under “Works with Google”.
Next is a screen with “Manage accounts” for the title. Under “Add new”, we will see [test] The Smart Thing Train (or whatever you choose when setting your Google Action project display name). Select it and watch the OAuth linking happen before your eyes.
After selecting [test] The Smart Thing Train, a small authorization screen pops up. Select “Accept” and the various passes of OAuth will happen automatically between the Express.js and Google servers.
Disclaimer: I’m only going through the passes for OAuth and not actually implementing any of the security checks. For example, “Authorize” screen would be where a user would input his or her username and password. It’s ok though, because our train is made of steel.
Meet Jimmy. This bright guy never gets tired of responding to my endless commands. It’s like he has his own source of energy. Oh, I think I forgot to mention, he’s a light bulb.
So how did this linking process actually work? What did the Express.js server do? What did the Google servers do? If you don’t care about OAuth and just want to get to commanding your little voice butler, go ahead and skip the rest of this section. If you do care, here we go.
When I selected [test] The Smart Thing Train, Google sent a GET request to https://d8a236bd.ngrok.io/authorize. The Express.js then served up a static HTML page that presented the “Authorize” screen to my phone. Once that screen was opened in my phone, Google added redirect_uri and state parameters to its URL.
Upon pressing “Accept”, a POST request is sent from the Authorization page to https://d8a236bd.ngrok.io/login, which is supposed to check login credentials (I didn’t use any) and then redirect to the redirect_uri supplied in the previous step along with the state and a special code that I generated.
Next Google sent a POST request to https://d8a236bd.ngrok.io/token with the OAuth client ID and secret I set in the Google Actions console as well as that special code that was generated. I like living on the wild side and didn’t confirm any of this in the Express.js server. As part of the OAuth standard, I sent back an access_token, refresh_token, expires_in, and token_type. Google then supplies these tokens in future HTTP requests to the Express.js server.
If you’re still curious, you can google OAuth and find plenty of material about how it should be implemented to correctly provide security. What we did here was enough to git-er done and linked! Check out the source code for the OAuth route handler I used if you’d like to see the implementation.
Looks like we’re coming up on our next stop…
Stop 2: Handling Google Smart Home Intents
Welcome to Stop 2: Handling Google Smart Home Intents. Here I’m going to build the engine that runs the action. Smart home actions must be able to handle four different intents from Google’s servers.
- SYNC
- DISCONNECT
- EXECUTE
- QUERY
You can sort of deduce what each of these intents mean by their names, but how do we handle them? Remember that fulfillment URL at https://d8a236bd.ngrok.io/fulfill? Yeah that one. That’s where we put this engine to work.
SYNC and DISCONNECT
The /fulfill route handler uses the Actions on Google Client Library, which can be installed via npm. This route handler is located in the Express.js repo here. I’ve implemented two intents to start with: (1) the SYNC intent and (2) the DISCONNECT intent. The SYNC intent has a JSON request body of the following form that hits the /fulfill route.
{
"requestId": "8e0fa320-138d-4454-bbcd-f143f8851432",
"inputs": [{
"intent": "action.devices.SYNC"
}]
}Instead of processing these requests directly in the handler, I hooked up the the Actions on Google Client Library, which provides a set of functions that accept callbacks for each of the intents.
In this route handler, I’ve included an isUserLinked flag in the cache to keep track of when I connect and disconnect my Google account. The smarthome library from actions-on-google is then instantiated and exported under module.exports.fulfill. To handle the SYNC intent, I just pass a callback that accepts the request body and headers into the onSync function.
The SYNC intent is what gets initiated immediately after OAuth account linking. This is your chance to tell Google what devices you have as well as their types and traits. I won’t go too deep into their architecture, since you’d probably prefer to hear that from the horse’s mouth. Daniel Myers from Google did an awesome writeup that goes into plenty of detail.
Inside the onSync callback, you’ve probably noticed our good friend Jimmy the light bulb! His type is action.devices.types.LIGHT and he has a trait of action.devices.traits.OnOff. Nothing too crazy on this train.
If you go back through the process to link up our Express.js server and stop at the “Manage accounts” page in your Google Home app, you’ll notice [test] The Smart Thing Train under Linked services. If you select it, you’ll see an option to “Unlink account”. If you’re feeling dangerous, go ahead and unlink and that will fire off the DISCONNECT intent.
In the Express.js server, the onDisconnect callback just sets the isUserLinked flag to false. The usefulness of this will be more obvious later… If you did this, don’t forget to relink! I’m going to add some more coal to the fire soon.
EXECUTE and QUERY
Ok, so I’ve got Jimmy linked up and welcomed to my home. He’s a cheerful guy and wants to brighten up my day even more. The only problem is he’s forgotten how to turn off and on. I guess he’s gonna to learn today.
The request body for an EXECUTE intent is pretty intense. Fortunately Google has some great documentation here. This is where we finally get to handle a voice command like “Hey Google, turn off Jimmy”. Immediately after the EXECUTE intent, comes a QUERY intent (docs here) to find out Jimmy’s state. This also gets triggered if you ask about Jimmy. For example “Hey Google, is Jimmy on?”. Ok, enough talk. More code.
From the https://github.com/jcartw/google-your-smart-thing repo, checkout the 2-handle-intents branch. Here’s a set of terminal commands to get our Express.js server fired back up. Make sure to spin down your previous one. Also, if you restart ngrok, you’ll need to update your URLs in your Google Action project.
$ git checkout 2-handle-intents
$ yarn install
$ yarn startstarting server on port 9000
Now we’re ready to put Jimmy to work. On your Google Assistant device (e.g. mobile phone, Google Home, or Google Nest), go ahead and say “Hey Google, turn on Jimmy” and watch the magic happen. Isn’t that just deeply satisfying? Maybe I need to get out more… Well for now I have Jimmy and code to keep me company. Jimmy is curious about how the EXECUTE and QUERY intents were implemented. Let’s look.
There’s now an onOffState variable that has been added to the route handler cache. Here’s is where Jimmy’s state is held for now. There are also two internal functions that are used to manipulate and query Jimmy’s state: (1) _actuateLight({ deviceId, turnOn }) and (2) _getLightOnOffState(deviceId).
The EXECUTE intent is handled with a callback inside of the onExecute function. Inside this callback, I’ve added a bunch of code to traverse through the request body and find a command with the name of action.devices.commands.OnOff. Additionally, this command has an “on” flag as its parameter, which the Express.js server uses to determine whether to turn Jimmy off or on. Then I used _actuateLight to carry out the command.
It’s a little simpler to implement the QUERY intent. As you’ve probably guessed, that gets handled in the onQuery callback where we use the _getLightOnOffState function and package up a response according to Google’s API. Ok, so now we’ve got something to play with, but this isn’t ready for public launch yet. We’ll have to implement the Request Sync and Report State portions of Google’s API. These are used to keep Google’s HomeGraph database in sync with Jimmy’s state as it changes. But right now we don’t have any other way of updating Jimmy’s state!? That’s going to change in our next stop.
Stop 3: Jimmy gets an Upgrade
Welcome to Stop 3: Jimmy gets an Upgrade. Jimmy is tired of living in the confines of our Express.js server memory. He wants to get his own place and feel the wind in his wires. But where should he move? He could go to the neighborhoods of ThingsSpeak, Iotery, or ThingsBoard. Well, Jimmy loves getting amped up in the mornings and found out that there’s a great juice bar in downtown Iotery. So I think his decision has been made.
I explained in a previous article how to setup an Iotery account and use its smart light bulb emulator.
Here’s the super short version. Go to https://iotery.io and make a free account. After that go to https://emulator.iotery.io/smart-light and click the “Connect” button and then the “Still need to configure iotery?” link and follow the guided steps.
In the Iotery dashboard, I’ll update the device name. Jimmy is too shy to correct people who forget his name, so I’m going to do it for him.
Ok! Now let’s update the Express.js server to be able to communicate with Jimmy again. Run the following commands in the terminal.
$ git checkout 3-upgrade-jimmy
$ yarn installI’m going to make a file called .env.development in root level of the repo (the same level as the .gitignore file) to store some environment variables. Add the following text to that file and update the variables with your actual credentials. You can get your device UUID and team API key from the Iotery dashboard.
export USER_ID=WHATEVER_USER_ID_YOU_WANTexport DEVICE_ID=YOUR_DEVICE_UUIDexport IOTERY_TEAM_API_KEY=YOUR_API_KEY
Now type this into your terminal to fire this engine up.
$ yarn run start-with-envstarting server on port 9000
The Google Home app needs to be unlinked and relinked now that the device ID has changed and we don’t have Request Sync implemented yet. After successfully relinking, we can send Jimmy a message over the internet. Say “Hey Google, turn on Jimmy” and boom! Would you look at that.
The Express.js server didn’t need too many updates either to dial into Iotery. You’ll notice that the onOffState variable is no longer cached in memory in the server. The main updates were made in the _actuateLight and _getLightOnOffState functions. Not bad.
Ok, now that Jimmy is all grown up and independent in Iotery-land, I’ll be able to implement Request Sync and Report State. Those are coming up on the next stop.
Stop 4: Request Sync and Report State
Welcome to Stop 4: Request Sync and Report State. These steps may at first seem unnecessary, but they are critical for keeping Google’s HomeGraph database in sync with Jimmy’s state. Plus Google won’t accept an action for public launch if they aren’t implemented. Like my momma used to say… my roof my rules ¯\_(ツ)_/¯.
Request Sync
Anytime information about your device changes, such as its name or if another device is added to your system, you’re supposed to send out a Request Sync request. This causes Google to send out a SYNC intent request to your server.
Now since the Express.js server is going to be sending a request to Google HomeGraph, it needs an API key from Google. Follow the steps on this page https://developers.google.com/assistant/smarthome/develop/request-sync. Look under Enable the API and download credentials in order to get your API key.
Once you’ve got the HomeGraph API key, update your .env.development file to include it.
export USER_ID=WHATEVER_USER_ID_YOU_WANTexport DEVICE_ID=YOUR_DEVICE_UUIDexport IOTERY_TEAM_API_KEY=YOUR_IOTERY_TEAM_API_KEYexport HOMEGRAPH_API_KEY=YOUR_HOMEGRAPH_API_KEY
Next checkout the 4-1-request-sync branch and restart the Express.js server.
$ git checkout 4-1-request-sync
$ yarn run start-with-envSo what we want is to be able to send a Request Sync from the Express.js server to Google whenever Jimmy decides he’d like to change his name. Fortunately, Iotery provides webhooks for all of its REST routes and I happen to know which one corresponds to updating the device name. Go to the Iotery dashboard and create a webhook action for Update Device. Now I’ll set the webhook address to https://d8a236bd.ngrok.io/update-device.
Jimmy is feeling a bit more formal today and asked me to start addressing him by Jim. I’ll just go into the Iotery dashboard under his device page and update that.
I see a couple of logs in my server terminal now. I’m going to check the Google Home app on my phone. Yes! His name is now Jim. That was easy enough. Now it’s time to implement Report State.
starting server on port 9000
WEBHOOK: /update-device
SYNC intent
Request sync successfulReport State
Google’s smart home interface includes both audio and visual elements. Since they want to be able to display the state of all connected devices FAST, they require that third-party systems report the state of their devices anytime it changes. This makes it so they don’t have to wait on a bunch of QUERY intents to return for all the devices in the house before taking an action.
If you look at the docs for Report State you’ll see that Google requires that you report state after every SYNC, EXECUTE, and QUERY intent. Plus anytime the state changes outside of these events.
In order to send a Report State request, you need to obtain a Service Account Key. Follow the docs at https://developers.google.com/assistant/smarthome/develop/report-state under the section Implement Report State to get your key.
Take your key (it should be a JSON file with fields like project_id, private_key, and client_id) and rename it to google-action-key.json and place it in the google-action folder of the server.
Let’s get the version of the server with Report State implemented up and running. Switch to the 4-2-report-state branch and spin up the server.
$ git checkout 4-2-report-state
$ yarn install
$ yarn run start-with-envstarting server on port 9000
We can actually toggle Jim’s state directly from https://emulator.iotery.io/smart-light. Upon each toggle, the emulator uploads new state data to the cloud. So, I’m going to add another webhook to capture this update and use that in my server to send out a Report State request. I’ll make a webhook action for Post Data and set the webhook address to https://d8a236bd.ngrok.io/device-data.
Let’s see this guy in action. I’m going to toggle Jim off in the emulator.
Looks like the server console is showing some action…
starting server on port 9000
WEBHOOK: /device-data
Reporting state...
Successfully reported device state
On/Off state: OFFAnd there you have it! Now everybody can talk to Jim and know whether or not he’s having a dark day. Let’s check out the route handler source code now and see how far we’ve come. There’s now a handleDeviceUpdate route handler that sends out a Request Sync anytime Jim’s info is updated within Iotery. We’ve also got a _reportLightOnOffState function that sends out a Report State request and gets called in the SYNC, EXECUTE, and QUERY intents as well as the /device-data webhook. And finally we see that the isUserLinked flag gets used to prevent the server from sending out Request Sync and Report State requests if the user isn’t linked.
We’re coming up on the final stop now.
Final Stop: Wrap Up
That was quite the ride. I hope you enjoyed passing time with Jimmy (AKA Jim) as much as I did. We went through setting up a Google Action project, OAuth linking with an Express.js server, setting up an IoT device with Iotery, as well as implementing all the Google Smart Home intents (SYNC, DISCONNECT, EXECUTE, and QUERY). And let’s not forget all the crazy times we had with Request Sync and Report State.
Come back and ride again!
Until next time.
