Homebrew Hue-compatible smart lighting that won't cost you a fortune: a hardware-centered tutorial.


For those of us who enjoy occasional weekend soldering...
  It went like this: I stumbled upon this video as I was just about to renovate my new apartment, and was immediately captivated:


 I'm usually skeptical of all things "smart home" - I want my home to stay dumb and cozy - but being able to compose different scenes out of RGB lights to set the mood sounds great! 

  This is where I started to be on the lookout for smart lighting every time I went to any home improvement store. After all, when it comes to light bulbs at your home, local availability is a high priority, right? The picture was disheartening: only warm white light bulbs were readily available most of the time.

  Remembering the magic word "DIY" in the project name, I decided to dig deeper and found out that diyHue unifies both DIY and ready-to-use lights. What that means is you can make any kinds of lights using inexpensive off-the-shelf components, and if you get sick of "making" approach you can add some store-bought lights to the mix!

  diyHue project is relatively well-documented, but I had to stagger through some parts largely by trial and error due to my imperfect knowledge of this area. Another pitfall: there's a repository devoted to the lighting controllers and switches that you can make, but it's far from complete. 

  That's why I had to prototype and design my own PCB to control RGBCCT (RGB+Warm white+Cool white) light strips, which I'll be sharing here. On my way there I'll try and do a sort of a deeply detailed tutorial, just to help anyone with basic soldering skills replicate my setup.

Final demonstration of my diyHue setup at home (sound omitted intentionally 🙂) :

0. Links

1. Preparations

  Smart lights don't just work after you plug them in - there's some groundwork to be done first! Fortunately, it's only two things: running the Hue gateway emulator code and installing at least one of the scores of Hue mobile apps.

1.1 Gateway

I won't delve into the gateway setup process as it's: a) well-documented, and b) more along the lane of generic IT knowledge rather than hardware. DIYHue documentation is your friend, the whole "Getting started" section is a detailed description of how to get the gateway with emulator code running.

Easy way out: run docker image on your PC

> Suitable for a quick trial run of the system.
I ran the image once on my laptop as a proof-of-concept, then proceeded to run the same image on an SBC (single board computer). Note: the images expand upon clicking.


The official Hue app won't find the gateway during scanning unless you specify the correct MAC!
This is where you get the MAC address...

... and the IP address for docker call above.

More permanent setup: dedicated hardware

The most popular SBC at the time seems to be Raspberry Pi, but in fact any will do. I eventually settled on a BeagleBone Black - the random pick from my parts drawer.

Pick any.


Of course, the board has to be protected against elements and dust if you're serious about this endeavor. My BBB was fitted out with a 3D printed enclosure (thingiverse link) : 

Gateway enclosure
Safely tucked away in a cupboard under the stairs TV, neighboring the WiFi router and Airsonic server.

The docker container that you run is the same, although the command needs some additions as it's going run 24/7:

... this time the MAC has to be obtained via command line, e.g.: 

1.2 Mobile apps

  • The official Philips Hue app is worth having, but at times it's not really up to the task (e.g. a sudden gateway update from Philips locks it until the corresponding version in diyHue is bumped).

  • My another top pick would be the Hue Essentials application. It's a third party app and seems to perform better in some ways (like, I still don't get how to set up the remote control in the official Hue, but here it's easy-peasy).

2 Lighting

Here are the basic components of every smart lighting setup, most of these I'll be covering below:

----
Light bulbs, LED strips, remotes, remote controlled mains sockets and wall switches.
----
  This list is, of course, highly subjective. Your setup may include only some of those, or some extras, but I consider these things to be essential for any serious and permanent smart lighting arrangement.

The best way to think of smart lighting is as the addition to the regular lighting.

I cannot stress the point above enough - if you expect yourself/your relatives to open up the app and fumble with the toggles on the screen each time they want to use restroom - well, don't fool yourself. Instead, think of smart lights as a cool addition to your lounge zone: imagine coming home after a day of work, kicking back on the sofa, putting on some nice music plus some fancy lights.

Short glossary:
RGB (red, green, blue) - marking on lighting equipment (generally on LED strips) meaning it's capable of displaying all possible colors of rainbow.
RGBWW, RGBCW - same marking as above, with dedicated white LEDs added (you can combine red+green+blue to obtain white, but it's usually not as bright). CW = cold white, WW = warm white.
RGBCCT = RGBWW, RGBCW, i.e. RGB lighting with 2 dedicated sets of white LEDs, both warm and cold whites.

Now you know everything it takes to make an informed decision about which type of lights you want where! I personally went for RGBCCT when possible.

Let's dive into each of of the light types I've outlined above:

2.1 Light bulbs

Most cheap WiFi-controlled light bulbs actually have ESP8266 with programming pins exposed inside. 
This RGBCCT light bulb for instance https://www.aliexpress.com/item/32920447431.html is the one I ordered at random intending to make it Hue-compatible:

This is what came in the mail. Incorrectly marked as RGBCW, whereas it's got the full spectrum.


Popping open the globe-shaped top cover reveals two PCBs

The two PCBs we can see above are: LED carrier board with warm white(W), cold white(C) and RGB LEDs; and another board with ESP8266 controller. It's the latter one we're after, now I'll fetch something sharp and carefully extricate the LED ring:



This operation, in turn, reveals the programming pads: RX, TX, GND and GPIO0:



... corresponding to the following connections (NOTE: the serial adapter has to operate at 3.3V - mine's got the switch between 5V and 3.3V but there are fixed 3.3V versions on the market as well) :



Now comes the tricky part: soldering some temporary wires to the pads on ESP8266:


Closeup shot

When this is done, the reflashing process can commence! Keep in mind that ESP needs to be powered up during flashing, meaning I'll have to plug my Frankenbulb into the mains.

This is the most dangerous part of the process! Repeat at your own risk. 
To alleviate the danger of electrocution, I found a lighting fixture that you can unplug from mains (see picture below). Then I screwed in the ready and connected flashing gizmo when it was unplugged, and only afterwards plugged the lighting fixture into the wall.



This is the sketch we want to flash:

Tip:  each light has a name like "Generic RGBCCT light". Once you'd made more than one light, this quickly gets out of hand. I used https://github.com/a-type/adjective-adjective-animal to generate light IDs:

and paste it here:



Flash the sketch, disconnect GPIO0 from GND and cycle the mains power. You'll notice the bulb broadcasting a new WiFi network with your lightName. Connecting to this network automatically opens up this page in your browser (if it doesn't, just go to 192.168.4.1) :  



Fill in the credentials for your home WiFi network (note: only 2.4GHz networks are supported) and hit Save.

Now it's time to check if the bulb connected to your home network. This step is optional but I like to do that just to be sure. Connect to your home network again and scan it for ESP hosts:


The command above should yield at least one line with the IP address (I got 5 because my network is chock full of ESPs). Opening this IP in your browser proves the light has connected to the home network:



Another thing to note: flashing the previous sketch has added the ability to program over air:



.. which means we can unplug the abhorrent flashing rig, desolder the temporary wires and attach the LED ring back again, finally returning to the safe zone! You should be able to add the new light in Hue application now and play around with colors:



  The catch is, the colors are going to be mixed up! (unless you're incredibly lucky). There's no way to tell which Arduino pins are hooked to which color channel, but fortunately there's not that many usable pins on ESP8266: https://randomnerdtutorials.com/esp8266-pinout-reference-gpios/

  According to the table from the link above, there are only 7 pins that are A-OK to use as outputs (0, 2, 4, 5, 12, 13, 14). Knowing that they're unlikely to have used GPIOs 0 or 2 as they're also used to select boot mode leaves us with just 5 pins: 4, 5, 12, 13 and 14 - which is exactly how many you need to control 5 color channels!

  The remainder of our job is an exercise in guesswork and repeated flashing. In my case I simply had to switch warm white and cold white channels:



Checklist before gluing the LED ring back in:
  • The light bulb is visible in Hue app, and you can control it: change colors, turn on and off.
  • All light channels are assigned correctly, e.g. when you set the color in app to red, green, blue, warm white and cool white you see exactly that color on the light bulb.

Gluing the LED ring dead to the frame...


And voila, you got your own Hue compatible RGBCCT light bulb for next to nothing:


The process I've outlined above is easy to do in batch, saving your time and money.

2.2 LED Strips

  I've been searching far and wide but couldn't find a schematic for RGBCCT ESP8266 controller in open access (probably not far an wide enough, because to me it sounds like an obvious thing to make if you're into DIY). So I had to spin my own and share it.

  The ultimate goal of this is to have the PCB that anyone can order in bulk and always get the consistent result. The point of prototyping? If I'd ordered the PCB made according to my initial schematic, it simply wouldn't work. You fish out every little mistake in your schematic while making a prototype, and proceed with PCB based on that.

  Of course, I went through this torment so that you wouldn't have to - skip straight to the PCB section below to make your device with minimal hassle.

Breadboard

  There are several variations of ESP8266 derivatives, the most commonly used being ESP-01 and ESP-12, plus the Arduino-like dev boards: NodeMCU, WEMOS D1 etc.

  Since we're starting out by prototyping, it'd make more sense to go with one of the latter (link to what I eventually bought) - but I hate to wait for the slow boat from China and I didn't prepare in advance, so  ESP-12 was my choice - abundant in my parts drawer and, unlike ESP-01, with more than one spare GPIO (link).


  One doesn't need a lot of electronic parts to get started on prototyping: prototyping board with an optional enclosure if you intend to use it later (thingiverse), ESP8266 controller, 5 NPN transistors and 5 N channel MOSFETs (one per each output channel), 3.3V regulator (or a buck voltage converter - this is more efficient, later I switched to those in all of my boards), misc. connectors, resistors, capacitors, etc..



Get started by assembling the power distributing part of the schematic: incoming 12V connector, 3.3V regulator and just power . You'll note I had to resort to "wiry legs" for my ESP - a time that I could save if I used a devboard like NodeMCU. Power it up as an initial check - in my experience, most soldering errors are actually shorts between power lines. You should see LEDs blinking once on your ESP upon bootup.
Only power distribution + ESP soldered on.

Check #2 - if power distribution works and ESP powers up, add programming headers and verify if you can program ESP with a Blink sketch. ESP8266 can be programmed with a simple 3.3V serial adapter. (link) 
Same as above plus programming headers.

Voila! Animation below shows the protoboard at a later stage, almost ready to go, but this is essentially what you should see after uploading a sketch - microcontroller happily blinking away:



Let's proceed by soldering on the remainders of our allotted components. When you're done, the switching circuit can be tested with that same Blink sketch on different pins, connecting any 12V LED strip to the outputs.


The final sketch that we want to flash on this board:
https://github.com/diyhue/Lights/blob/master/Arduino/Generic_RGB_CCT_Light/Generic_RGB_CCT_Light.ino

I've added a filtering capacitor on 12V rails for a good measure, but I don't think it was essential. My enclosure had an empty compartment anyway, so the cap filled that in nicely!



The box that I've printed wasn't really tailored to my needs, and general laziness prevented me from adjusting it BEFORE the print, therefore I had to melt some holes with soldering iron. My principle: reliability and function first, sleek look later. ☺



To-melt list: aperture for the power supply cable + wide slot for LED strip wires:
1) Cut a hole in the box 2) Put your junk wire in that box!

Aaand we're done! 
Ready to use prototype.

PCB

  There's one PCB but two variants of the enclosure: regular and even waterproof! Waterproof is a choice for lighting up you bathroom or outdoors. ☺

  This time we're surely going with ESP-12 (link): it's got exactly 5 spare GPIO pins, and it's the most widely used one of the lineup.

  You've ordered the parts from the BOM, ordered the PCBs, what now? A prime time to design a 3D printed enclosure for them, that's what!

This has been the most advanced modeling enclosure design exercise for me so far:
  • Generate an STL model based on the board's dimensions with Ultimate Box Maker (covered in Hackaday a while back where it captured my attention):


  • Make use of extended integration between KiCAD and FreeCAD called Step Up (link1, link2)

  • Model the whole device (enclosure + PCB + components) to check if the enclosure is a good fit for the board:

  • Print it!
Fresh from under the nozzle :)

When the long-awaited package arrived, I was ready to populate the PCBs.
Front + back shot of the PCB, just arrived.
  
  After two uneventful weeks full of rosin and rubbing alcohol fumes, I had my 5 specimens of RGBCCT controllers:
Completely assembled board.

Some more vanity shots:
Completely assembled board. Front+back shot.

4 out of the 5 PCBs I've ordered.


  Time to bulk-flash the firmware! (same file as we flashed to the breadboarded version):




The board can be attached to those tiny posts on on of the parts:



The final step is soldering the wires to the power connector and screwing them in:
Solder the wires to the connector first, then connect to the board, minding the polarity.

Internals of the device when it's ready:



  As I've mentioned before, there's a waterproof version of that enclosure, completely modeled from scratch in KiCAD (full_assembly.FCStd contains both the assembly and editable models - stark distinction from e.g. Autodesk Inventor where each model inhabits one file, and assembly is yet another file). 

  KiCAD StepUp once again helps us model the final look of the device and check for collisions:



  The assembly process is a bit more involved in this case. Leaning on my previous experience in waterproofing custom 3D printed enclosures, I've decided to smear the insides in epoxy and dab some silicone on the "lip" of one of the parts as a gasket of sorts. Plus throw in some desiccating agent - the most readily available such agent is an ordinary silica gel bag.

  Okay, you got the two parts of the enclosure printed (any filament will do, including PLA). Have some disposable gloves and something to cover the table ready:



Apply copious amounts of epoxy to the insides of the printed enclosure:

For the next waterproofing endeavor, grab some ordinary silicone in a tube:



Smear a layer on the  protruding "lip"-thingie (at least that's what I'm calling it for the lack of a more proper name) and let it all cure for several hours.
It's even better if you can apply the silicone around the holes, but if you do cover them (like I did here) it's still not a problem.

Now you'll want to plug the waterproof connectors:

About to close it all off....

  We're ready to seal the innards off from the environment. Since we're not doing that in a controlled manner (air we breathe is always humid to some extent) I got a trick up my sleeve to combat that - throwing in a bag or two of regenerated gel:


Place the regenerated bag inside the enclosure and screw the two parts of the enclosure together with M3 bolts and nuts.



 The surfaces coming together will push some silicone out, effectively forming an impenetrable seal: 



Trim the edges of the silicone gasket so that it looks decent:



Final product:

UPDATE 2021-01-30:

The 3.3V regulators were heating up quite considerably. Makes sense as they're dissipating the difference between 12V and 3.3V (at whatever current is consumed) in heat. That got me thinking about the fact the controllers are powered up around the clock, so I replaced all 3.3V regulators with cheap buck converters from China:
Any small step-down converter from China will do.

  This cut the current consumption in half. Nice! Although not really what I'd call great savings: 45mA at 12V is a mere half of a Watt: 

12 V * 85 / 1000 A = 1.02 W (power consumption with linear regulator)
12 V * 40 / 1000 A = 0.48 W (power consumption with buck dc-dc regulator)
1.02 - 0.48 W = 0.54 W saved.
Before the modification: 85mA idle current consumption After the modification: idle current down to 40mA
Is this modification worth it? Well, it's up to you. In the meanwhile, I'll try and design another version of the circuit board with DC-DC regulator built in.

2.3 Individually addressable LED strips (WS2812)

Same as LED strips, but can be controlled over 1 wire. The downside is that they're all RGB - don't expect white or cold whites.

Breadboard


What we'll need:

Improvised parts list.


As per usual, we'll get started on the power tract and check if it works as expected (converts 5V to 3.3V, that is):




Proceed by adding the programming headers and the ESP, and flashing the venerable Blink sketch:



  There's just one step that separates this electronic junk from being useful - solder on the LED connector.
  Don't be confused about the fact that mine has 5 pins - you only actually need 3, but it was the only polarized connector I found in the parts box.



Ready device:

Arduino sketch to be flashed:

https://github.com/diyhue/Lights/blob/master/Arduino/Generic_WS2812_Strip/Generic_WS2812_Strip.ino

One note: ESP01 can't be programmed over air -  not enough memory. Make sure you always have easy access to programming headers.

Check whether it works by flicking the colors:

PCB

 Turns out you don't need to go though all this to have a ESP8266 controlled WS2812! They already sell the ready-to-use modules designed with pretty much the same schematic as the DIY version above: [link]
 

To be continued

2.4 Remote controls

Breadboard

Fortunately, the schematic for the battery-powered diyHue remote exists:


... and even the PCB is available in that same repo! The catch is, it's in the Gerber (manufacturing) format, so it's effectively set in stone. Don't know where to find those exact same parts the author has been using? You're outta luck.  

Anyway, the schematic is all good, time to see if it holds water (figuratively, please don't dunk your breadboard) :

any Lithium battery, any PNP transistors, LDO regulator, any (TP4056 here) battery charger, 4 buttons, ESP and various resistors & capacitors.


As per usual, solder in and check the essential parts first:

Seeing if the battery can be charged - you can never trust the modules costing 10 cents apiece:



Then soldering on the voltage regulator:



... putting in the controller now:



Firmware link:


In my specific case, I haven't been able to make it work right off the bat:



Having composed a small button press testing sketch, I found some issues with my soldering (namely bridges and poor quality of joints). Sharing the sketch here in case someone faced the same:




The fruit of my labors looked REAL ugly until I found a proper box for it:



I always go by one rule I've established a long time ago: a trashy enclosure out of a lunchbox for a prototype is better than none at all:






PCB

I'd make one if I ever needed more than one! Besides, everyone seems to like the otherworldly and surreal look of that lunchbox on feet.

2.5 Remote controlled mains sockets

Prototype

  This one turned out to be so simple that the picture below simultaneously contains the parts needed AND the assembled schematic:

The wires leading off the camera are going straight into the AC.


Okay, if you insist:



  The reality turned out a bit harsher than I expected, because V+ pin on the relay board is wired into the linear regulator, therefore it can only accept voltages higher than 5V. But nothing is unfixable, right? Simply desolder the volrage regulator and short Vin to Vout:



  Why did I use the timer relay, you'll probably want to ask? Hue doesn't have any timing constraints on the lights - you can flip the light on and off 10 times a second for all it cares. However, if we were to plug the incandescent bulb into such a socket, it'd quickly deteriorate. The timer component of the relay limits the switching rate: set it up so that it clicks the relay a second or so after you've sent the command.

Firmware link:


The enclosure? How about a cheap timer wall plug from the supermarket?



Cram it all in, and see to the insulation between the components (I threw in a piece of a plastic bag, ha-ha). 



But seriously, what you're seeing on the pictures here can only be considered a proof of concept - leaving it in the wall continuously puts you in a real danger of combustion every second.



Let's see if it works now - I've always dreamed of turning the salt lamp into a part of a smart lighting ecosystem.



Finished prototype:



Now, a ready version of that same device that won't burn your house down



I splurged on the real devices and bought an actual suitable enclosure for them: that's something I do very rarely!

Following the same schematic, we'll add a fuse before the power supply, leaving the remainder pretty much the same:



The PCBs are fastened securely to the walls, too!




Closeup shot. The only design choice here that I resent is epoxy I've used to fix the power supply to the enclosure. Anyway, the only real downside here is I'll have to break it out of the box when it's FUBAR:



The end result looks unassuming, yet it's much safer: safe enough to leave unattended in your apartment.



3 Installing every single thing in its rightful place + my setup







One controller at the bathroom:






... three controllers for each of the shelves at the common room, hidden inside smartly inside the kitchen island:













... and a bonus one inside the tech cupboard, lighting up the undersides of it!




4 Troubleshooting
I've had this setup working round the clock since summer of 2020. During all this time some unpleasant things naturally cropped up, so I'll describe how to cope with those:
  • Power outages: when the power is restored, WiFi connection takes longer to start up than ESP8266 lights, which means the lights go into their "unprovisioned" mode, broadcasting WiFi network instead of joining the yet-inexistent one. Solution: cycle the power to ESP8266 controllers (in other words, unplug them from mains and plug back in). Alternatively, join each WiFi network, open 192.168.4.1 in your browser and click the Restart link (sometimes it's not even there, so you have to go to http://192.168.4.1/?reset in your browser). Rinse, repeat for each light.

Comments

  1. UnknownApril 9, 2022 at 5:18 PM

    You sir, are hardcore! Nicely done.

    ReplyDelete
  2. AnonymousMay 10, 2022 at 12:04 PM

    Nice https://shortn0tes.blogspot.com/2020/03/diyhue-smart-lights-tutorial.html

    ReplyDelete
  3. AnonymousSeptember 10, 2022 at 11:05 AM

    Thank you of your work, hard work pays off :)

    ReplyDelete

Post a Comment