Brave New World

Lets just come out and say it.

This sucks. All positive social media posts and celebraty love messages don’t change the fact that this really sucks. Quarantine, social isolation, the constant stream of ever growing case numbers, and the constant fear of the unknown. It really sucks.

This is a really hard time for everyone. Humans have evolved to be social creatures, and that sense of social connection is built into our mental well being. We develop physical symptoms when cut off from the group. We derive self identity, value, and purpose by our place within that group.

Social isolation is painful for humans to endure. It affects us deep down in the ancient part of our brain that developed when we were huddled in caves and scraping together necessities for survival among much larger and terrifying predators. Our sense of community is what helped us build family structure and evolve a sense of belonging to the tribe. Hell, it probably developed when we were small mammals scurrying around dinosaurs, huddling under rocks.

Our social evolution is what allowed us to build communities, towns, cities, countries. This modern world has evolved to cater to our need for social bonding and community. We connect to people around the world at the speed of light, over radio, fiber, copper and even paper for the intimate touch.

This is why our current world is scary and alien to us. It’s robbing us of something so fundamental to our species we forget about it when its here and cant explain the problem when we’re gone. We take our sense of community for granted, so much so that we dont even realize how important it is. How deep the wiring in our brains goes. We biologically need other humans, our small tribes and families. Coronavirus is slowing eating away at that community, forcing us to isolate, to break up the binds. I don’t think its COVID that scares us, or the possibility of getting sick, even severely. I think its the isolation that is fundamentally hurting us, deep where we’ve forgotten it, in the core of what makes us human.

Don’t get me wrong. I fear for my children’s health, my wife’s, my parents and my friends. I take every step possible to keep my loved ones safe. But I’m starting to feel the psychological damage isolation causes us. It’s like a cancer, it attacks us where we’re most vulnerable in a slow and insidious way.

I’m very fortunate right now. I’m with my children and my wife. My tribe is with me and we’re safe. That’s important. But there are those out there cut off from their tribe, isolated alone. It’s a difficult position for humans to be in. We need others, even strangers. So please, reach out to those in your circle who may not have others with them. A phone call, a web meeting, even a letter if you’re the romantic type can help dispel sense of being alone and adrift at sea.

Our world is not the world we evolved in. The community may be damaged but it doesn’t have to be broken. Use the tools we have to keep it together during this period. Who’s the one person in your circle who lives alone, maybe hasn’t seen or heard another voice in a while. Call them. Let them know they’re still part of your tribe.

I don’t talk about this publicly, and only some of my tribe knows. But I struggle with sever depression that I manage through medication and the support of my family. I can tell you from my personal experience that this isolation is very difficult one those of us with mental health battles. I cant speak for everyone obviously, but it anxiety and panic episodes have led to many sleepless nights. I’m not expressing this for sympathy or likes. I am sharing this to help people understand how vulnerable some of us are to the erosion of our social community. So please, be patient with those of us who suffer quietly and may be affected in different ways.

We will get through this, together as a community and as a world. The world may be different on the other side, but we’ll have our tribes and our community back. Help each other. Be kind to each other. We’re in this together.

FTDI-Gate: The WRONG way to fight counterfeits

NOTE: This article was written in 2016 for an internal group. I’ve decided to publish here for posterity
What or Who is FTDI?

FTDI is a major developer of USB communication chips and drivers. Their most common product is a line of serial to USB chips and drivers. If you have a USB to serial converter or a black box that connects older equipment to a USB port, chances are there is an FTDI chip at the heart of it.
FTDI also provides drivers for their chips. A driver is a piece of code that runs on your computer that tells Windows how to talk to the device.
The FTDI line of USB chips are incredibly popular and common in many electronic widgets. They are found in test equipment, DIY products (Arduinos, BeagleBone’s, Raspberry Pi etc) as well as many consumer electronics. Anything that speaks USB and Serial probably has an FTDI chip somewhere.

The Problem When You’re The Best:

FTDI chips are so reliable and popular that the market is awash with counterfeit FTDI chips. These fakes look just like genuine FTDI chips and can be impossible to distinguish without drastic testing. Often the only way a consumer can tell is by replacing a buggy piece of hardware with a different vendor. The real problem with the counterfeit chip business is that the end user or even a mid level producer often has no means of knowing if the chips they are getting are genuine or not. The supply chain is so heavily saturated with counterfeits, even distributors have a difficult time guaranteeing genuine parts.

FTDI’s Answer to Counterfeiting – FTDI-Gate Part 1:

In late 2014, rumors and blog posts started appearing describing problems with an updated FTDI driver being pushed out through Windows Update. Although not documented by FTDI, it was soon discovered that this new driver would modify the counterfeit chip in a specific way, rendering it useless. In the community this is known as “bricking” the device. The driver would alter a parameter in the chip that would stop it from functioning with any other computer. This was FTDI’s answer to the counterfeit chips, to disable them without the end users knowledge or consent.
As one could imagine, the online community erupted against FTDI. This action was seen as duplicitous, manipulative and possibly illegal. An analogy to this would be Honda remotely destroying your car because they discovered a counterfeit part in the engine. Punishing the end user for market problem far upstream was a very heavy handed action by FTDI who were forced eventually to rescind the damaging driver update, although bricked devices were not repaired.

FTDI Strikes Again – FTDI-Gate Part 2:

Fast forward to February 2016. Stories began to pop up about strange, random behavior of serial to USB adapters. With the stink of FTDI-Gate still fresh in peoples memories, the builder community began to suspect FTDI of shenanigans once again.
Like a rerun of a bad sitcom, it was soon determined that FTDI had once again modified it’s drivers, pushing a new poison pill out through the Windows Update service. This time, if the driver detected a counterfeit chip, the driver would inject text into the data stream. Specifically the driver would inject “NON GENUINE DEVICE FOUND!” into the serial and USB data streams.
This is different than bricking the fake chip, and in some ways worse. Imagine for a minute you have an old chiller that communicates to a BMS system via a serial to USB converter, which is not such an uncommon occurrence. The chiller receives commands and sends data via this serial to USB converter. Suppose again that this serial to USB converter was purchased from a major retailer like Best Buy, and was manufactured in China. The changes of having a counterfeit FTDI chip is probably 60/40. How would the chiller respond if it suddenly received “NON GENUINE DEVICE FOUND!” from the communication line? How would the BMS system respond to receiving this string from the Chiller? There really is no way of knowing.
Injecting invalid data into a production system is akin to sabotage, resulting in unpredictable results. Bricking the device at least leaves one in a known state.
This firmware update has not yet been revoked.

So Now What?

I wanted to bring this story up because it’s a good illustration of the types of struggles with electronic systems and why they suddenly stop working. In this case, there really is nothing the end user can do to ensure they are getting genuine FTDI parts other than being particular in where they buy products. Also as a warning to everyone who uses USB to Serial converters. If you happen to one day run into problems with the converter, you may have been affected by the FTDI-Gate.

Persistent Remote Access on JCI NAE’s

So this was a fun little project that was born from some security research I was doing on the Johnson Controls NAE product (more on that here) and wanting to try out my latest toy, the Hak5 Bash Bunny. As far as I know, this vulnerability effects NAE’s from version 9 and below.

First, a bit of housekeeping. This research was conducted by me alone and is not in anyway associated with my employer. The conclusions, opinions and views expressed here are mine and do not reflect those of my employer or any other group. All research was conducted on equipment I had authorization to access. At no time were any systems or cloud services I did not have authorization to access involved in this research.

Second, some history. I discovered this vulnerability in June while conducting research for another project, linked above. I informed the Johnson Controls Product Security team of the vulnerability and potential exploitation. They responded that the vulnerability had been removed in newer version of the NAE and would not issue a public disclosure, which I suppose is fair.

Automatic Login Vulnerability

The NAE prior to version 9 is essentially a Windows XP Embedded computer in a fancy enclosure. Underneath that fancy enclosure is a VGA monitor port and if you plug in a monitor you can see the boot process. During that boot process after Windows XP starts, the NAE automatically logs in under the MetasysSysAgent account with a cmd window open. There’s a delay of 10 or 12 seconds before the account locks itself and you’re left with a standard Windows XP lock window.

I’m not sure what purpose this auto-login and lock process has but it provides an interesting opportunity to exploit. During that 10 second window armed with a monitor and a USB keyboard, you can do pretty much whatever you want. This can include changing the MetasysSysAgent password or creating a new admin account on the box. I started thinking of an attack scenario around this where an attacker would use this 10 second delay to plant some sort of RAT (Remote Administration Tool) or other program to gain control of the box. It would be pretty difficult for even a really great Social Engineer to talk his or her way into access to these boxes while carrying a keyboard and monitor. Enter the Bash Bunny

BashBunny
Bash Bunny by Hak5. Please don’t sue ūüôā

Exploitation

Now the Windows XP installation itself is pretty stripped down. There’s no Telnet, no FTP and the IE installation seems restricted to not be able to download any external files. My Windows XP knowledge is admittedly limited so I couldn’t think of a way around this restriction. Having recently received my shiny new Bash Bunny I started experimenting with its various attack modes, such as functioning as a USB thumb drive and a keyboard.

I set up the Bash Bunny to copy netcat and reg.exe (a command line registry editor) to the windows box, and then add a registry entry to automatically start netcat on reboot. The netcat session would reach out to my Command&Control server on the web with a shell session. So, by monitoring the netcat port on the C&C server, I would get a cmd.exe session from the NAE under the MetasysSysAgent user account.

I have a video showing this attack below. I apologize for the quality of the video, it was shot last minute. Also, I used the term RCE (remote command execution) in the video to describe this attack. In hindsight, I’m not sure sure if that’s accurate but you get the idea

Scenario

From the video, you can see how quick this is to execute. So a plausible scenario for this attack would be to social engineer your way into accessing the box for a quick 90 seconds, plug in the Bunny and reset the device. From then on, you would have remote persistent access to the BAS network, which is pretty valuable. There’s all sorts of other options here also.

Conclusion

Protecting physical access to resources is obviously very important to infrastructure security. The problem in the BAS industry I think is that these devices are typically installed with other building infrastructure that regularly requires inspection. In most office buildings or commercial properties, people trust a man in a button up shirt and a clip board. I know from first hand experience how easy it is to move around a building by looking like a contractor. Getting quick access to an electrical closet or comm room isn’t all that difficult in these facilities. 90 seconds later, you have a persistent presence on the network.

Metasys ADS/ADX/NAE Login Authentication Bypass

I wanted to provide a write up of my research relating to the security vulnerabilities Johnson Controls published on August 15th in order to clear up some confusion and better describe the actual attack chain. These vulns has been assigned CVE-2019-7593 and 7594. Also tracked under the DHS ICSA number ICSA-19-227-01. I recommend reading the official notification provided by Johnson Controls here also (PDF).

First, a bit of housekeeping. This research was conducted by me alone and is not in anyway associated with my employer. The conclusions, opinions and views expressed here are mine and do not reflect those of my employer or any other group. All research was conducted on equipment I had authorization to access. At no time were any systems or cloud services I did not have authorization to access involved in this research.

Second, some history. I began looking into the Metasys around November of 2018. My original goal was to investigate whether I could convince the NAE to load a modified software image. As I started looking into the process the Metasys UI and the target system interact, my attention shifted to the login procedure and how it was secured. I disclosed my findings to the Johnson Controls security team around May of 2019. My findings were accepted and a 90 day embargo was requested to put the appropriate notice together etc. I found the JCI security team to be perfectly reasonable in their requests and professional in their dealings with me.

Now on to the fun stuff. I’ll explain briefly how the Metasys login process works, followed by a description of the 7593 and 7594 vulns and finally a proof of concept of how I exploit these vulns into an attack. It’s important to be familiar with how RSA public/private key encryption works. This information is relevant for Metasys version 9 and below only. Anything newer then that, your guess is as good as mine.

Metasys Login Process

The Metasys UI is a Java application (which I’ll refer to as the Client) that is downloaded directly from either the ADX, ADS or NAE (the Server) being connected to. Whether this is done directly or through the JCI Launcher is inconsequential. When the Java app is started, there is a process of initialization and initial resource loading. During that time the Client requests the Server’s public RSA key. Once the initialization is complete, the Client prompts the user for a username and password.

getKey
HTTP Post request to Server requesting RSA public key

The password is hashed (using SHA1) and a credential package is constructed using the plaintext username and hashed password. This credential package is then encrypted using RC2. The encryption key for the RC2 payload is encrypted using the Server’s RSA public key. This forms the completed payload which is sent to the Server using an HTTP POST request.

login
Login request sent to Server

I’m assuming at this point the Server decrypts the RC2 encryption key using its RSA private key, then decrypts the username / password with the RC2 key. The Server will compare the hash and username to a security file containing a list of valid users. If the login credentials are valid, the Server will respond with a Session Token encrypted with the same RC2 encryption key.

successful
Server sending successful login response with Security Token

All subsequent requests between the Client and Server are accompanied by the same encrypted Session Token. Without a valid Session Token, requests are rejected.

Vulnerabilities

Common RSA Keys CVE-2019-7593

As described above in the login process, it’s simple to request the public RSA key from the Server using a simple POST request. Ideally, the RSA keys should be unique for each Metasys device. However I was able to test 10 NAE, NCE and ADX devices and found they all returned the same RSA public key. Therefore the RSA key pairs seem to be common among these devices. If the RSA private key is compromised on one device, it is in effect compromised on all of these devices. An analogy to this would be having all cars of the same make/model being keyed with the same ignition key. This is not ideal. However due to the next vulnerability, this turns out to be pointless anyway.

Hardcoded RC2 Encryption Key CVE-2019-7594

The Metasys UI doesn’t seem to randomly generate the RC2 encryption key used to encrypt the login credentials and other sensitive information. The key is hardcoded in the Java application, which is downloaded to the users computer. It’s very simple to decompile the Java application using something like the Java-Decompiler (which is an awesome tool) and located the hardcoded RC2 encryption key. The fact that the RC2 key is hardcoded, easily available to the user and never changes renders the use of RSA public/private key encryption useless. This is because the RSA key is used to encrypt the RC2 key only, but we already know the RC2 key so we can skip the RSA problem entirely.

decrypt login creds
Decrypting captured login credentials using fixed RC2 key

Other Vulns

The acute reader will notice the hashing algorithm used for the password is problematic. SHA1 is known to be a cryptographically weak algorithm that is routinely broken. I have made liberal use of Crackstation in the past, which has proven very effective against SHA1 password hashes. However as I’ll show in the attack chain below, you don’t need to decode the SHA1 hash. Also, these hashes are not salted.

Cracking the base64 decoded password from above.

Exploit

The exploit I used in this case is essentially a replay attack. Valid encrypted login credentials are captured using your favorite method and then used to login to the BAS system. Normally you cant login using a hash and would need to decode the password. However I can show below that you can replay the hash and login without being able to decode the password.

Capture Login Package

Capturing the valid login session takes a simple Man-In-The-Middle attack. This can be done a number of ways. Compromised router, public wifi, compromising the users computer etc. To simulate a MITM attack I used the fabulous and fantastic Lan Turtle from Hak5. This is a great tool that deserves it’s own post. So without going into detail, the Lan Turtle is acting as a MITM, specifically looking for the login packet and sending the details out to my Command and Control server.

With the encrypted login request in hand, we can use the hardcoded RC2 key to decrypt and recover the plaintext username and hashed password. If the password can be cracked using something like Crackstation, then the job is done. If the password is complex and cant be cracked, there’s one more step to finish the replay attack

Proxy Replay

Obviously you cannot just type the password hash into the login screen and presumably the password is complex enough to resists hash cracking.

My solution to this was to create a proxy relay server that I used to connect to to Metasys. This proxy would relay all traffic between the Client and Server, but would inspect the requests for the login request. When the login request is seen, the Proxy would replace the username/password I entered with the stolen username/password hash. This would be considered valid by the Server and I would be logged in under the stolen user account.

In this example, the Server is at 192.168.1.10, the Client is at 192.168.3.218 and the proxy relay at both 192.168.3.60 & 192.168.10.229.

Proxy Relay Example

The Proxy Relay listens for an incoming connection and relays traffic to the target Server, searching for the login request

Proxy Server replacing login request with captured credentials

In the Wireshark capture below you can see traffic flowing from the Client through the Proxy to the server. Notice the highlighted packets showing the incoming login request and the login request being sent to the Server. Notice the encrypted strings are different, despite using the same encryption keys. This is because the login credentials entered have been replaced by the captured credentials, resulting in a successful login under the compromised user account.

Login credentials being replaced with compromised account

Conclusion

There are few occasions where hardcoded passwords/keys/secrets are a good idea. This is not one of those occasions. It should have been relatively easy for the Java application to randomly generate a new RC2 encryption key with each transmission (sending the key protected by the RSA encryption). Also, I don’t see why the RSA keys have to be common to each device. The RSA keys could be generated when the system is installed, or even on system reboot. These vulnerabilities go to show how important it is to lock down network access to a BAS system, drawing the ring of trust as tight as possible.

Hope this was helpful to anyone reading. If you have questions or would like more information, feel free to contact me.

Note: You’ll notice I’ve neglected to include any actual plaintext keys. This is intentional.

Organizing Memories

So many memories we’ve forgotten

When was the last time you had pictures printed? How many pictures do you have on your phone or tablet? Do you even remember what pictures you have anymore or where they’re from?

My wife’s grandmother passed away 2 years ago. Her grandfather, John, and his wife were prolific picture takers and frequent vacationers. Over their last 5 years or so they had amassed several thousand pictures on various devices. Most of these were on two iPad’s, the balance being spread across a few CDs and usb flash drives.

So the problem was presented by the family to me. We have all these pictures, we don’t know what is a duplicate of what, we don’t know what’s on these drives or the ipads but we need it all organized in a single folder on a new laptop. The intent was for John to be able to view and curate these pictures on a laptop (that I would purchase and set ip), which was something he was struggling with on the ipad.

In a very rare act of taking the high road, I’ll skip my usual hate rant for Apple products, their apparent hatred for their user base, and their obvious goal to frustrate those of use who serve as the family tech support.  Suffice to say that I managed to invoke the right magic, spoke the correct incantations and beckoned the  appropriate powers to get all the pictures off of both ipads and safely onto my machine. We’ll skip the pain and move on.

All told, I now had around 2600 pictures to work with and I had two objectives in mind.

  1. Eliminate any duplicates in this collection of pictures
  2. Sort this collection of pictures in to a folder structure for the years and months the pictures were taken.

Finding Duplicates

This was an interesting problem to solve. I am very sure this wasn’t the most efficient way to solve the issue but it was rather interesting to me. My approach was to read through the various folders for the picture files. For each file, I load the filename, extension and path into a database. I also take an md5 hash of the picture as well as the date the picture was taken as reported in the exif data. These additional data points are also loaded into the database.

For those unaware, an md5 hash is a unique fingerprint of the data within the file, which for my purposes is enough to uniquely identify the file based on its content and not on its file name. Also, exif data is data saved inside a picture file the identifies several properties of the picture such as the camera it was taken on, the camera settings and the date the picture was taken.

From here, weeding out the duplicate pictures was a trivial SQL query to select the unique md5 hashes. Now I thought that was pretty neat. See the code for this first stage below. You can see that I am using exiftool to pull out the exif data. Its a great tool written by Phil Harvey.  Also below is the end result of this script

if [ $1 -z ];
then
echo "Need to include the root directory containing pictures"
exit
fi
if [ $1 = '-h' ];
then
echo "   [-d]:to create duplicate database "
exit
fi</code>

search_dir=$1
#echo $search_dir
first=0
find $search_dir -iname '*' -print0 | while IFS= read -r -d '' entry
do
if [ $first -gt 0 ];
then
filename=$(basename "$entry")
extension="${filename##*.}"
path=$(dirname "$entry")
md5hashString=$(md5sum "$entry")
md5hash=($md5hashString)

imageDate="$(exiftool "$entry" |
grep -m 1 "Date/Time Original" |
sed "s|Date/Time Original : ||g" |
sed "s|:|-|g")"

if [ -z "$imageDate" ];
then imageDate="$(exiftool "$entry" |
grep -m 1 "File Modification Date/Time : " |
sed "s|File Modification Date/Time : ||g" |
sed "s|:|-|g")";
fi;
if [ -z "$imageDate" ];
then
imageDate="1980-01-01"
else
imageDate=($imageDate)
fi

sql="INSERT INTO Pictures (Path,File,Extension,Hash,Date) VALUES ('$path', '$filename', '$extension','${md5hash[0]}','$imageDate')"
mysql -D GGPics -e "$sql"
fi
first=1
done

Picture hashes

Organizing the Final Product

Once I had everything in the database, I made a new table of distinct hashes, essentially weeding out the duplicate files. Next was the organizational step which was rather trivial. For this, I wrote a small python script (again probably not as efficient as it could be) to accomplish the organization. The script reads through the unique table, determines if a folder exists already for the year the picture was take, and the month the picture was taken. It creates the folders as needed and then copies the image into the new structure. Code and results below

oldPath = result[1]
oldFilename = result[2]
oldDate = result[3].strftime(‘%Y-%m-%d’)
oldFile = open(oldPath + "/" + oldFilename, "r")

#see if year folder exists
if not os.path.exists(newLocation + "/" +oldDate.split("-")[0]):
os.makedirs(newLocation + "/" + oldDate.split("-")[0])
month = calendar.month_name[int(oldDate.split("-")[1])]
#see if month file exists
if not os.path.exists(newLocation + "/" + oldDate.split("-")[0] + "/" + month):
os.makedirs(newLocation + "/" + oldDate.split("-")[0] + "/" + month)
if((oldDate.split("-")[0] == "1980") or (oldDate.split("-")[0] == "1900")):
newPath = newLocation + "/" + "UnknownDate" + "/"
filecount = len(os.listdir(newPath))
newFile = open(newPath + str(filecount+1) + ".jpg","w")
else:
newPath = newLocation + "/" + oldDate.split("-")[0] + "/" + month + "/"
if os.path.isfile(newPath + oldDate + ".jpg"):
filecount = len(os.listdir(newPath))
newFile = open(newPath + oldDate + "_" + str(filecount+1) + ".jpg","w")
else:
newFile = open(newPath + oldDate + ".jpg","w")
newFile.write(oldFile.read())
newFile.close()
oldFile.close()
lastID = lastID+1
print newFile.name
print str(lastID) + " of " + str(totalFiles) + " done"
curs.execute("UPDATE DistHash SET Processed=1 WHERE ID=" + str(result[0]))
conn.commit()

At the end I end up with a structure that looks something like this. I don’t have the actual pictures anymore so its showing 0 files but you get the idea.

Picture Structure

All in all, this was a pretty fun project I intend to work on a little more and use for my own organization. My wife is a prolific picture taker and over the years I have multiple backups. So having this type of tool to quickly organize and weed out duplicates is pretty useful.

More importantly, this tool gave me the ability to give something important to John. It let me give him back his memories. So often we leave our memories locked up in our devices. I was pleased to be able to give John access to the pictures he hadn’t seen in a long long time and the memories that are attached to them. It reminded me of how precious and fragile our memories are.

“Daddy, lets build electronics!” – How a book changed my life

This is less an interesting technical blog post and more just a retrospective on my past and what it has given me today.

As long as I can remember I’ve loved electronics, since the first time I took apart a tv remote when I was 12 (much to my fathers unhappiness). Looking at the green epoxy coated circuit board, it made me think I was looking down at a green landscape, resistors and capacitors dotting the scenery like towns and villages. Then I was just an observer discovering new worlds. Eventually in highschool I began learn electronic components and design, to which I am always grateful to Mr.Langil my electronics teacher. Now I felt like a creator of landscapes, not just an observer. I was hungry for knowledge and without the easy access the internet provides us, I turned to books.

Sometime in 2000 I picked up a copy of “Engineer’s Mini Notebook: Timer, OpAmp & Optoelectronics Circuits & Projects” from Radio Shack.¬† This was the first I had ever seen of this book series¬† or its incredible author, Forrest Mims. Initially I had been interested in it because I had worked with 555 timers in the past and Op Amps were like a new magical power waiting for me. But after looking at this small book I was captivated by its hand drawn diagrams and notes. There was a romantic, real quality that I couldn’t resist. This wasn’t a text book, this was like a hand written note and treasure map directed to me. I can’t say how many hours I spent with this book or how many of these circuits I built, but this book took me from a dilettante to someone with a real passion to build.

Fast forward through highschool, two rounds through university and a decade and a half in various careers. Through the wonderfully deterministic world of digital electronics, the seductive magic of analog signals, and finally programming and embedded systems where I could finally communicate with something.  Now I spend most of my time writing code and less time building hardware but now and then will still pull out my old kits and put something together.

My 5 year old daughter spent a week this summer at a STEM camp. During that week she built a simple flasher circuit out of conductive play-dough, a battery and an LED. She loved this project and was so excited to show me, knowing her daddy is an engineer. She told me all about how it worked, how the electricity moved through the metal and the dough to make the light work. She wanted to build more and decided that we would build electronics together.

My home office is littered with containers of components, half finished projects and books. I started scrounging around for bits and pieces to put together while my daughter picked through my bookshelves. It may have been the colorful cover that drew her attention but for whatever reason she picked up this same “Engineer’s Mini Notebook” that had caught my attention so long ago. She asked if I had drawn the diagrams and pointed out to me the LED and resistor symbols she had learned.

I would be lying if I said I wasn’t a little emotional seeing her get so excited over this same book that had started me on my journey.

I picked a simple project that would appeal to her and the next day we were down at the local electronics shop to pick up the bits I was missing. An hour of breadboarding later, my daughter learning to use a multimeter to measure capacitors, and we had our first project. She loved it and wouldn’t stop playing with it. Breadboards are temporary and this was a memory I wanted to last. So the next day while she was at school I soldered it up on a prototype board. And here it is, our first Father Daughter project.

First Project

This is the Toy Organ project from page 23 of Forrest Mims’ “Timers, OpAmps & Optoelectronics” entry from the Engineer’s Mini Notebook series. And it still sits on the window sill of my daughters room. Every now and then when she’s in her room I can hear the tones from the circuit.

I can’t express how grateful I am to Mr.Mims and his book series. It played an important role in my life and is now helping me introduce my daughter to the exciting adventure of world building. We’ve already picked the next project to build, maybe one that doesn’t make so much noise this time. I look forward to many weekends spend with this book on one side, and my daughter on the other.

Geographic Lookup From IP Address

Grepping through Log Files

Managing a few small web sites and some web applications, I’m constantly finding myself grepping through log files looking at access activity¬† and suspicious entries. I’m always interested in looking up the geographic location of where access requests are coming from. Is the traffic local or am I getting hit from Russian or Chinese IP addresses? There are a lot of online tools to get geographic information from an IP address, such as here and here, buts its time consuming when you want to look up lots of addresses. So I wanted a script to do more of a bulk lookup.

IP lookup API’s for scripting goodness

While looking around the internet for a good, free API for geographic lookup, I found ipstack, a free IP Geolocation API service. Signup is required in order to get an API key, but the free version should give you most of the functionality needed, with 10,000 requests per month. The API is pretty easy to use also, and a simple curl request can do the trick. The API returns the results by default in JSON format.

In this example I’m making a curl request to the ipstack api, requesting the geographic location for Cloudflair’s 1.1.1.1 DNS server.¬†The response comes back in JSON so I’m piping the output to jq, which is a really handy JSON parser for bash.

curl http://api.ipstack.com/1.1.1.1?access_key=xxxxxxx | jq

The ipstack API has different options to only request some of the data fields and not others. I’ll leave it to the reader to look them up if interested.

Parsing Log Files for feedstock

So this was a really good start, but I want to script multiple requests to the API and store the results in an sql database for easy lookup. I’m a big fan of using databases for information storage and usually opt for using sql even if its a bit overkill.

Apache is generally pretty good at keeping log files. The default apache config keeps aceess logs in /var/log/apache2/access.log , where errors are kept in /var/log/apache2/error.log. This may be different on your system, so RTFM. These files are nothing if not dense but they are reliably formatted. Awk is your friend here. You can use awk to pull out the log file IP addresses. From there, the rest is easy peasy.

sudo awk '{ print $1 } ' /var/log/apache2/access.log | sort | uniq

I’m piping the results of awk through sort an uniq, which sort the results numerically and strip out duplicate results. If you have multiple log files you could always cat them together before sending them through awk.

Now the magic happens

Now we want to run through each IP address, make the request from ipstack and store the geographic results. Its possible to do this purely with bash,but my bash game isn’t quite that strong. I chose to write a small Go program to do the job.

package main

import (
"database/sql"
"encoding/json"
"fmt"
_ "github.com/go-sql-driver/mysql"
"io/ioutil"
"net/http"
"os"
)

type Result struct {
IP string
Continent_name string
Country_name string
Region_name string
City string
}

func main() {
var myResponse Result
args := os.Args[1:]

if len(args) != 1 {
panic("Need an arguemnt")
}
target := args[0]
fmt.Printf("Using target %s", target)
url := "http://api.ipstack.com/" + target + "?access_key=xxxxxx&fields=ip,continent_name,country_name,region_name,city"
resp, err := http.Get(url)
if err != nil {
panic(err)
}

defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
err = json.Unmarshal(contents, &myResponse)
if err != nil {
panic(err)
}

//connect to database
db, err := sql.Open("mysql", "ipLookup:@/IPTrack")
if err != nil {
panic(err)
}

stmt, err := db.Prepare("INSERT INTO Results (IP,Continent, Country,Region,City) VALUES (?,?,?,?,?)")
if err != nil {
panic(err)
}

_, err = stmt.Exec(myResponse.IP, myResponse.Continent_name, myResponse.Country_name, myResponse.Region_name, myResponse.City)
if err != nil {
panic(err)
}
db.Close()
}

Remember, my standard Code Caveat applies.

You can pipe the output of the awk command above to this program, which will load the ipstack api lookup results to sql. What you’re left with is a handy database full of geographic locations of your visitors. Pretty neat.

This makes it really clear this particular server is getting a lot of attention from China. This is good information to now make a decision as to whether I want to impose a geographic firewall rule or not. I’ll leave that up to the reader to decide.

BACnet MSTP

A brief Description

Here is a brief explanation of how BACnet MSTP works. This will be somewhere between an “ELI5” and something approaching a Coles notes. FOr a technical description, I would suggest picking up the BACnet Specification, or at least checking¬† the BACnet Website.

A BACnet MSTP (BCMSTP from now on) consists of at minimum a Coordinator, a Master and a pair of wires connecting them.

A Coordinator, as the name implies, coordinates all activity on the MSTP trunk. It keeps track of all communication statistics, as well as correcting any disruptions in the communication. Typically the coordinator also manages any communication with external devices or a larger IT network. Coordinators are normally larger devices running bona fide operating system.¬† An example of a coordinator would be a Johnson Controls NAE , which runs Windows XP Embedded (no I am¬† being serious) or Tridium JACE, which run RIM’s QNX OS.

A Master is any device on the BNMSTP network that can talk and share information. In the context of Building Automation, a Master is a device that has sensor inputs and outputs for controlling equipment. The Master encodes real world conditions such as temperature, humidity, current draw, pressure and shares that information with other Masters on the trunk. The Master also has outputs for controlling mechanical or electrical equipment, such as fans, heaters or boilers. Masters can typically be loaded with software for controlling this equipment on its own through programming.  Masters are often smaller 16 or 32 bit microcontroller devices. Some examples of Masters would be the Johnson Controls FEC, or the Reliable Controls Mach Pro

The BACnet MSTP Trunk should look something like this, where Node and Master are interchangeable for our purposes. For the time being, ignore the Server, we’ll get to that later.d

 

The Communication Process

Unlike the chaotic and haphazard communication of its UDP cousin, MSTP communication is quite an orderly affair.  Communication happens in a predictable, almost deterministic way.

There are some ground rules for MSTP to keep in mind

1)Each master or node on the network has an address between 1 and 254. The Coordinator (also known as a Supervisor) has address 0.

2)Communication packets on the MSTP bus are defined as frames. There are 7 defined frame types which are described below.

Frame Types

Frame # Frame Type Description
0 Token Pass Pass the token to another master
1 Poll for Master Sends out Probes looking for a new master not yet identified
2 Reply to Poll for Master Responds to frame type 1.
3 Test Request Diagnostic Use
4 Test Response Diagnostic Use
5 Data Expecting Reply This frame contains data the requires a response from the destination. Used to request status information
6 Data Not Expecting Reply This frame contains data that does not require a response. Used to respond to requests in frame type 5.
7 Reply Postponed Used to respond to frame type 5 if the master is too busy

Each master on the network keeps a few specific pieces of information recorded.

MyAddress = the masters address.

NextMaster = the next masters address in the bus, defaults to 0 on startup.

MaxAddress = the highest address number allowed on this network. This is set in programming. Lets assume its 64 for this example.

Communication Flow

A simplified exchange would look something like this.

In the beginning all devices come on line. The network is unmapped, and none of the masters are aware of each other yet. The coordinator begins sending out polls for a master starting at address 1 until a master responds.

The coordinator continues sending out polls to incrementing addresses until a response is received. This tells the coordinator who the next master in the network is.

The token is passed to the next address. Since this master does not know who the next master is, it starts polling for the next master. This process continues from one master to the next as the network is mapped.

Eventually the token is passed to the last master on the network. This master will start polling for the next master up until the Max Master is reached. This signifies to the master that it is the last in line and it passes the token back to Address 0, the coordinator.  Now that the network is mapped, the masters can start sharing information about devices on the network

In order to request some piece of information, one master sends a Data Expecting Reply frame to another. Encoded in the packet is the specific instance number of the point the master wants to know about and what aspect of that point (ie present value, units, name etc). The host master sends back the response in a Data Not Expecting Reply frame. We’ll discuss the technical specifics of how data is encoded in a later post.

This is a pretty quick and dirty explanation of how BACnet MSTP works, but it should shed enough light to make the process a little more clear. I hope to expand on the technical details a bit more in the future.

Daedalus

Overview

Daedalus is a device I built in 2010 as my final project to complete my Bachelors degree at BCIT. It is a BACnet MSTP device that allowed me to communicate with an MSTP trunk and pull information from devices on that trunk very quickly.

During that time I was working as a controls technician, working on building automation systems. The tool set I had to troubleshoot and monitor devices on an MSTP trunk were well designed but significantly flawed. These tools would interrogate an MSTP host for every input/output point it had, as well as any software points programmed. On medium to larger hosts, this could take minutes to update and get new readings. This caused significant problems when I was just looking for a single point, or trying to tune a PID loop and needed fast results.

The goal of Daedalus would be to overcome this flaw and allow me to directly interrogate a single or small group of points much faster. Daedalus would physically attach to the RS485 communication lines of an MSTP trunk and communicate as if it were a host on that trunk. Daedalus had a Bluetooth link that would communicate to a user interface running on  a laptop.

Unfortunately I don’t have as much detailed information on this project as I would like, as it was nearly 10 years ago, but I’ll share what details I have.

Details – Daedalus

The software behind Daedalus is a preemptive multitasking real time kernel, which runs on an HCS12 microprocessor. The kernel, written in C/C++ was designed by David Quarbach, to whom all credit is due. On top of this kernel are several drivers to communicate both with the BACnet MSTP trunk, and to a Bluetooth transmitter. There is also a CLI for interfacing with the device directly over serial.

The limited BACnet stack Daedalus uses was written by me, and implements the minimum functions in order to maintain communication on an MSTP trunk, and poll other devices for status information of their inputs or outputs.

Daedalus Version 1

Daedalus went through several evolutionary phases, some rougher than others.

A short lived Version 2

This was one of the last iterations of the hardware design. The final version included a modular plug to plug into the MSTP trunk.

Version 3 – Final

As you can see, many of these components are modular packages from Sparkfun. However the HCS12 chip itself had to be landed on breakout board from Dangerous Prototypes, which is an excellent company to do business with. This was the first time I had to work with landing an SMT chip, and that was an experience. Not having any reflow tools, I had to do my work with a hot air station.

Dont worry, I cleaned up the solder job after the initial landing

 Details РUser Interface

The user interface was written in C/C++ and provided basic control over Daedalus. The UI, which was based on ncurses, allowed the user to start/stop the information gathering process, specify the target to monitor, as well as collect diagnostic information on the MSTP trunk itself

I grew up watching Tron, and this is what I was told the future would look like

Results

Daedalus was a great success for me. Not only was I able to monitor devices on an MSTP trunk much faster than with my existing tool set, but it taught me a great deal about embedded RTOS systems, and the BACnet protocol itself. I was able to use Daedalus in my work for a while after the project was complete, and it was a great help.

So why the name? Well that’s my Wife’s fault. According to Greek mythology, Daedalus built the Labyrinth of Crete, which imprisoned the Minotaur. He has several other accomplishments to his name and is considered one of the first Engineers in Greek mythos. I’ve always had a romantic attraction to his stories, having wanted to be an engineer all my life. So when I needed a name for this project, my wife who has a deep love for all things mythology, made the suggestion. Since then, most of my projects carry a name of a greek or roman mythological figure.

What is BACnet?

What exactly is BACnet?

Simply put, BACnet is a communication protocol that has become very popular in the Building Controls industry. It is a protocol that building control equipment use to communicate with each other in order to encode real world measurement values (ie temperature, pressure, humidity) and relay commands. It is a method of encoding and sharing information, which I suppose you could say about all communication protocols.

I’ve always enjoyed working with BACnet since I was first introduced to it in 2004-ish in my previous career as a controls technician. At that time I was coming from a world of proprietary technology and found it refreshing to be exposed to an open and license free communication standard. I also found BACnet very readable and understandable with some basic understanding of electronics and some standard tools. We’ll get more into that later.

What do I mean by the standard being open and public? Well the BACnet specification is openly published and anyone can buy a copy and read it for themselves. Its a very readable document, and with it anyone can write some software or build some hardware to use the BACnet protocol without having to pay licenses or royalties.

This is my copy of the BACnet standard I got in 2004 or so. Its well used, dogeared and missing a cover. Long story

BACnet is maintained by the American Society of Heating Refrigeration and Air-Conditioning Engineers, and I will leave it to ASHRAE to explain their history and development of the BACnet Protocol.

There are two main flavours of BACnet I focus on, and that is BACnet IP and BACnet MSTP.

BACnet IP, as its name implies, uses an IP network to communicate and UDP as the transport layer. Why UDP? Well, UDP (analogues to mailing a letter)¬† has very little overhead and can be implemented on pretty simple hardware. TCP (analogues to making a phone call) on the other hand has more overhead to maintain connections and could be taxing on older, smaller devices. Keep in mind we’re talking about small 8bit or 16bit microcontrollers implementing these stacks back in the day before 32bit micros became cheap and plentiful.

BACnet MSTP (master slave token passing) uses RS-485 and is limited to smaller network sized. This is still popular on smaller devices or in applications with smaller network runs. Its very easy to develop and troubleshoot.

The first time I got really under the hood with BACnet was in university when I built a piece of BACnet MSTP diagnostic equipment. This was a really interesting project involving hardware design and writing a preemptive multitasking kernel on a 32 bit microprocessor. This was Project Daedalus, and also involved writing my own (limited) BACnet MSTP stack. I’ll write up more about Daedalus at a later date.

Working on Daedalus really gave me an appreciation for what I could do with BACnet and how easy it was to develop. Since working on Daedalus, I’ve started a new project, Icarus, which is the direct decedent of Daedalus but for BACnet IP.¬† More on that later.

This work has also opened my eyes to the security ramifications of BACnet, and just how easy it is to exploit and possible cause significant damage. We all remember how Mr.Robot was able to bring down Steel Mountain using an Raspberry Pi and access to a thermostat.

I’ll discuss the feasibility of this attack in a later instalment. Spoiler alert, its pretty realistic.

So that’s my quick and dirty run down of BACnet. In upcoming posts, I’ll go into further detail on my own work exploring BACnet and various projects that I hope will shed more light on this protocol and its ubiquitous use in the industry. Enjoy.