Linux Mystery: linux-vdso.so.1

Everytime I compile and link a Linux binary, I see a dynamically linked library called linux-vdso.so.1.

[austin@localhost]$ cat hello.c
#include <stdio.h>

int main(int argc, char *argv[]) {
    printf("Hello, World!\n");
    return 0;
}
[austin@localhost]$ gcc -o hello hello.c
[austin@localhost]$ ldd hello
	linux-vdso.so.1 (0x00007ffee39dd000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f91baaec000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f91bad03000)

The libc and ld-linux I know about, but who is this linux-vdso? There is a man page that describes it.

The  "vDSO" (virtual dynamic shared object) is a small shared library that the 
kernel automatically maps into the address space of all user-space applications.  
Applications usually do not need to concern themselves with these details as the 
vDSO is most commonly called by the C library.  

...

Why  does  the  vDSO exist at all?  There are some system calls the kernel provides 
that user-space code ends up using frequently, to the point that such calls can 
dominate overall performance.  This is due both to the frequency of the call as 
well as the context-switch overhead that results from exiting user space and 
entering the kernel.

The rest of this documentation is geared toward the curious and/or C library writers 
rather than general developers.  If you're trying to call the vDSO in your own 
application rather than using the C library, you're most likely doing it wrong.

Huh. Neat. We are bringing in kernel functionality into user space in the form of shared object.

The man page goes on to describe that making system calls is expensive because we need to do a context switch to the kernel and back and there are some system calls that could really just be implemented as user space functions and it would save us a lot of time.

This seemed suspicious to me. Isn’t the point of a system call to clearly distinguish between user space code and kernel code? What system calls am I now bringing into my user space?

Well it’s actually just four syscalls (on x86-64).

• clock_gettime
• getcpu
• gettimeofday
• time

To verify, I dumped the memory contents of the process where the vdso exists.

[austin@localhost]$ gdb -q ./hello   
Reading symbols from ./hello...
(No debugging symbols found in ./hello)
(gdb) b main
Breakpoint 1 at 0x1149
(gdb) r
Starting program: /home/austin/projects/elf-collection/hello 


Breakpoint 1, 0x0000555555555149 in main ()
(gdb) info proc map
process 132181
Mapped address spaces:

          Start Addr           End Addr       Size     Offset objfile
      0x555555554000     0x555555555000     0x1000        0x0 /home/austin/projects/elf-collection/hello
      0x555555555000     0x555555556000     0x1000     0x1000 /home/austin/projects/elf-collection/hello
      0x555555556000     0x555555557000     0x1000     0x2000 /home/austin/projects/elf-collection/hello
      0x555555557000     0x555555558000     0x1000     0x2000 /home/austin/projects/elf-collection/hello
      0x555555558000     0x555555559000     0x1000     0x3000 /home/austin/projects/elf-collection/hello
      0x7ffff7db4000     0x7ffff7db6000     0x2000        0x0 
      0x7ffff7db6000     0x7ffff7ddc000    0x26000        0x0 /usr/lib/x86_64-linux-gnu/libc-2.32.so
      0x7ffff7ddc000     0x7ffff7f49000   0x16d000    0x26000 /usr/lib/x86_64-linux-gnu/libc-2.32.so
      0x7ffff7f49000     0x7ffff7f95000    0x4c000   0x193000 /usr/lib/x86_64-linux-gnu/libc-2.32.so
      0x7ffff7f95000     0x7ffff7f96000     0x1000   0x1df000 /usr/lib/x86_64-linux-gnu/libc-2.32.so
      0x7ffff7f96000     0x7ffff7f99000     0x3000   0x1df000 /usr/lib/x86_64-linux-gnu/libc-2.32.so
      0x7ffff7f99000     0x7ffff7f9c000     0x3000   0x1e2000 /usr/lib/x86_64-linux-gnu/libc-2.32.so
      0x7ffff7f9c000     0x7ffff7fa2000     0x6000        0x0 
      0x7ffff7fc8000     0x7ffff7fcc000     0x4000        0x0 [vvar]
      0x7ffff7fcc000     0x7ffff7fce000     0x2000        0x0 [vdso]
      0x7ffff7fce000     0x7ffff7fcf000     0x1000        0x0 /usr/lib/x86_64-linux-gnu/ld-2.32.so
      0x7ffff7fcf000     0x7ffff7ff3000    0x24000     0x1000 /usr/lib/x86_64-linux-gnu/ld-2.32.so
      0x7ffff7ff3000     0x7ffff7ffc000     0x9000    0x25000 /usr/lib/x86_64-linux-gnu/ld-2.32.so
      0x7ffff7ffc000     0x7ffff7ffd000     0x1000    0x2d000 /usr/lib/x86_64-linux-gnu/ld-2.32.so
      0x7ffff7ffd000     0x7ffff7fff000     0x2000    0x2e000 /usr/lib/x86_64-linux-gnu/ld-2.32.so
      0x7ffffffde000     0x7ffffffff000    0x21000        0x0 [stack]
  0xffffffffff600000 0xffffffffff601000     0x1000        0x0 [vsyscall]
(gdb) dump binary memory vdso.so 0x7ffff7fcc000     0x7ffff7fce000
(gdb) q

It’s really just an ELF so file!

[austin@localhost]$ file vdso.so 
vdso.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=112feb4b14e301806a8eafdcdd804c88bfa191d8, stripped

And here are the functions, as expected:

[austin@localhost]$ objdump -T vdso.so

vdso.so:     file format elf64-x86-64

DYNAMIC SYMBOL TABLE:
0000000000000bc0  w   DF .text	0000000000000005  LINUX_2.6   clock_gettime
0000000000000b80 g    DF .text	0000000000000005  LINUX_2.6   __vdso_gettimeofday
0000000000000bd0  w   DF .text	0000000000000060  LINUX_2.6   clock_getres
0000000000000bd0 g    DF .text	0000000000000060  LINUX_2.6   __vdso_clock_getres
0000000000000b80  w   DF .text	0000000000000005  LINUX_2.6   gettimeofday
0000000000000b90 g    DF .text	0000000000000029  LINUX_2.6   __vdso_time
0000000000000b90  w   DF .text	0000000000000029  LINUX_2.6   time
0000000000000bc0 g    DF .text	0000000000000005  LINUX_2.6   __vdso_clock_gettime
0000000000000000 g    DO *ABS*	0000000000000000  LINUX_2.6   LINUX_2.6
0000000000000c30 g    DF .text	0000000000000025  LINUX_2.6   __vdso_getcpu
0000000000000c30  w   DF .text	0000000000000025  LINUX_2.6   getcpu

It seems strange that the output of ldd doesn’t show an .so file on the disk to dynamically load but this is coming from the kernel and disk files are more of a user land thing.

Let’s make sure we aren’t making the syscall by actually using one of those vdso functions.

[austin@localhost]$ cat hello-vdso.c
#include <stdio.h>
#include <sys/time.h>

int main(int argc, char *argv[]) {
    struct timeval t;
    gettimeofday(&t, NULL);
    printf("Seconds: %lu\n", t.tv_sec);
    return 0;
}

We can use strace to see what syscalls are being made. We should not see the gettimeofday syscall in this case.

[austin@localhost]$ strace ./hello-vdso 2>&1 | grep "gettimeofday\|write"
write(1, "Seconds: 1619316657\n", 20Seconds: 1619316657

We must be using the VDSO version. My next question is to see if I can force the syscall to happen. I could not find a GNU linker option to turn it off. You can turn it off system wide using various kernel options, but not “per application” at link time. I also thought linking statically would do it (since the VDSO shows up in ldd) but even that didn’t work. The kernel/glibc really want to make sure I’m using the optimized version!

I was able to do a junk hack to make it happen by statically linking and then using a hex editor to mangle the string __vdso_gettimeofday to make it think that the VDSO version was never loaded.

[austin@localhost]$ strace ./hello-vdso 2>&1 | grep "gettimeofday\|write"
gettimeofday({tv_sec=1619316899, tv_usec=828106}, NULL) = 0
write(1, "Seconds: 1619316899\n", 20Seconds: 1619316899

This was kind of a dumb experiment, but it was a good way to learn about how the kernel and user land interact in ways that most people don’t think about too hard.

A Chatbot for pfSense Firewall Logs

A side project I have been working on is a chat bot to stream pfSense firewall logs to a central chat server so I can view them without needing to log into the web interface. I am using Errbot, a Python chatbot framework. Errbot separates the backend chat server backend code from your bot functionality. This lets you write a bot and have it work for multiple backends (IRC, Slack, Mattermost, etc.). Bot functionality is implemented using a plugin API so it’s relatively easy to share your bot’s functionality with others. It’s surprisingly fun to write chatbots that you and your friends/coworkers can interact with.

Github link: https://github.com/austinkeeley/err-pfsense

To install, either clone the project into the Errbot plugins directory or run the bot command

!repos install https://github.com/austinkeeley/err-pfsense.git

Configure the bot to point to your pfsense syslog file and where to stream the messages (the default identifier) – this depends on your backend. For IRC, it’s a channel name.

!plugin config pfsense {'LOG_FILE': '/path/to/your/logfile', 'DEFAULT_IDENTIFIER_STR': '#bots' }

After it’s configured, start reading the logs with the bot command

!start log

Features:

• Supports the firewall and the DHCP log events
• Reverse DNS lookup to turn IP addresses in the logs into hostnames automatically
• MAC address vendor lookup for DHCP logs

Reverse Engineering FTL: Faster Than Light

My most recent reverse engineering project has been to learn more about the internals of the indie strategy game FTL: Faster Than Light.

FTL: Faster THan Light is a 2012 rouge-like strategy game in a science fiction setting. You command a crew of a ship tasked with outrunning a rebel military fleet and delivering a warning message to headquarters on the other side of the galaxy. Your movement between the stars is facilitated by the FTL drive, a science fiction device that allows faster than light travel.

The game plays like an RPG and a management sim with a combination of combat, decision making, and resource management. It’s an inexpensive indie title that is worth giving a try if you’re into strategy games.

It’s also brutally hard. It might be one of the hardest games I have ever played. This is why I wanted to do some RE on the game to try to cheat a little and finally see the ending.

The game has four resources: scrap metal, fuel, missiles, and drones. Scrap metal is the currency in the game. You get this from winning space skimishes, exploring abandoned space stations, and making deals with space pirates and shopkeepers. Scrap metal lets you buy new weapons, upgrade to your ship, and hire new crew members. Fuel powers your FTL engine and lets you make the jumps. Running out of fuel is generally a pretty bad situation since you’ll be stuck as the enemy fleet approaches. Missiles are ammo for weapons systems that ignore shilds and have potenial to do more damange than conventional lasers. Drones are an interesting game mechanic that can automatically fight alongside of you, perform repair duties, etc.

If you want to get right to the cheat engine I built, look at the project on GitHub: https://github.com/austinkeeley/ftl-cheat. It lets you max out your resources (see the screenshot) and it makes the game slightly more winnable.

Initial Notes

I’m using the Linux version purchased from Steam. Steam drops the game in the steam/steam/steamapps/common/FTL Faster Than Light/ directory. In this directory, you see the game launcher bash script FTL which just calls another script, data/FTL. This script checks your computer’s architecture and either launches the FTL.x86 or FTL.amd64 version.

There are no other shared objects and ldd just tells me that nothing else is dynamically linked in other than system libraries so we can assume that all the game’s code is in these binaries.

There’s also file called ftl.dat which I’m going to assume contains all the game’s data (graphics, music, story text and diaglogue, etc.). I’ll make a mental note to look for the code that loads this when the game initializes.

Digging into the Binary

I’m going to use the FTL.amd64 version. The first thing to notice is that the binary drumps a bunch of text to the terminal if you run it directly instead of through Steam.

[austin@localhost]$ data ./FTL.amd64
lib/SIL/src/sysdep/posix/time.c:82(sys_time_init): Using CLOCK_MONOTONIC as time source
Version: 1.6.9
Loading settings
Initializing Crash Catcher...
Starting up
Loading text
Initializing Video
Video: 1280x720, windowed
lib/SIL/src/sysdep/opengl/graphics.c:420(opengl_init): OpenGL version: 4.6.0 NVIDIA 390.116
lib/SIL/src/sysdep/opengl/graphics.c:421(opengl_init): GLSL version: 4.60 NVIDIA
lib/SIL/src/sysdep/opengl/graphics.c:422(opengl_init): OpenGL vendor: NVIDIA Corporation
###
### lots of OpenGL messages here
###
Video Initialized
Renderer: OpenGL version 4.6 (GL_VERSION: 4.6.0 NVIDIA 390.116)
Creating FBO...
Starting audio library...
lib/SIL/src/sysdep/linux/sound.c:168(sys_sound_init): Audio output rate: 48000 Hz, buffer size: 1024, period: 256
lib/SIL/src/sysdep/posix/thread.c:187(sys_thread_create): 0x6D1B70((null)): Requested priority 10 (actual -10) too high, using 0 (0)
Audio Initialized!
Resource Preload: 2.418
Initializing animations...
Animations Initialized!
Loading Ship Blueprints....
Blueprints Loaded!
Initializing Sound Data....
Generating world...
Loading achievements...
Loading score file...
Running Game!

That should give us a good idea of where to go after the entry point. There are a lot of strings in the binary that say things like:

/home/achurch/Projects/ftl/hg/lib/SIL/src/sound/core.c

Doing a little research, I discovered that is a debug string left by a software developer Andrew Church who ported the game to the iPad. I’m going to assume he also worked on the Linux port since the game uses an open source library he wrote, the System Interface Library for games (SIL). Having some of the source code is goign to make reversing the rest of the game easier.

As far as those strings dumped to the console, we can see the last one printed in the function at 0x00411ab0. We’ll call this one ftl_init. If we trace the call to this far enough, we can find the main function at 0x0040f390.

If I had to make a guess, I’d assume that there’s a data structure representing the game state (e.g. the crew, my ship’s name and health, the upgrades, etc.) and I can probably dump my memory and find it.

Dynamic Analysis

Let’s start with doing RE to determine how fuel works. This is easy because it decrements by one each time we do a FTL jump in the game.

I used memscan to find where my fuel variable is and then set a gdb watch to break when it changes so I could get a backtrace. Not surprisingly, it’s in the heap.

[----------------------------------registers-----------------------------------]
RAX: 0x0
RBX: 0x1dac0bc0 --> 0x801948 --> 0x4d57b0 (push   r15)
RCX: 0x0
RDX: 0x0
RSI: 0x0
RDI: 0x1dac0bc0 --> 0x801948 --> 0x4d57b0 (push   r15)
RBP: 0x1da726d8 --> 0x803a18 --> 0x51bf20 (push   r14)
RSP: 0x7ffde01a8a20 --> 0x1da726d8 --> 0x803a18 --> 0x51bf20 (push   r14)
RIP: 0x4c1d22 (mov    edx,DWORD PTR [rdi+0x700])
R8 : 0x1dec18c0 --> 0x1dff51e0 --> 0x1dfcc150 --> 0x45b0bb0 --> 0x0
R9 : 0x3
R10: 0x3
R11: 0x7
R12: 0x1da70d00 --> 0x1da70760 --> 0x0
R13: 0x1da74728 --> 0x804f00 --> 0x545cd0 (push   r12)
R14: 0x8
R15: 0x6d78 ('xm')
EFLAGS: 0x206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
[-------------------------------------code-------------------------------------]
   0x4c1d14:    mov    QWORD PTR [rsp+0x28],rax
   0x4c1d19:    xor    eax,eax
   0x4c1d1b:    sub    DWORD PTR [rdi+0x700],0x1
=> 0x4c1d22:    mov    edx,DWORD PTR [rdi+0x700]
   0x4c1d28:    mov    BYTE PTR [rdi+0x764],0x1
   0x4c1d2f:    test   edx,edx
   0x4c1d31:    cmovns eax,DWORD PTR [rdi+0x700]
   0x4c1d38:    mov    rdx,QWORD PTR [rdi+0x28]
[------------------------------------stack-------------------------------------]
0000| 0x7ffde01a8a20 --> 0x1da726d8 --> 0x803a18 --> 0x51bf20 (push   r14)
0008| 0x7ffde01a8a28 --> 0x54010c (jmp    0x5400c8)
0016| 0x7ffde01a8a30 --> 0x1da70d00 --> 0x1da70760 --> 0x0
0024| 0x7ffde01a8a38 --> 0x0
0032| 0x7ffde01a8a40 --> 0x1da70d00 --> 0x1da70760 --> 0x0
0040| 0x7ffde01a8a48 --> 0x8e0a27c47f837000
0048| 0x7ffde01a8a50 --> 0x1da70760 --> 0x0
0056| 0x7ffde01a8a58 --> 0x1da726d8 --> 0x803a18 --> 0x51bf20 (push   r14)
[------------------------------------------------------------------------------]
Legend: code, data, rodata, value

Thread 1 "FTL.amd64" hit Hardware watchpoint 1: *0x1dac12c0

Old value = 0x25
New value = 0x24
0x00000000004c1d22 in ?? ()
gdb-peda$ bt
#0  0x00000000004c1d22 in ?? ()
#1  0x000000000055adb7 in ?? ()
#2  0x000000000040fd64 in ?? ()
#3  0x000000000041332e in ?? ()
#4  0x000000000041398e in ?? ()
#5  0x000000000068e681 in ?? ()
#6  0x000000000040f5b0 in ?? ()
#7  0x00007fd28b7d7b6b in __libc_start_main (main=0x40f390, argc=0x1, argv=0x7ffde01ad508, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffde01ad4f8) at ../csu/libc-start.c:308
#8  0x000000000040f9aa in ?? ()

Here we can see the fuel being decremented by one in the instruction sub DWORD PTR [rdi+0x700], 0x1. I repeated the same steps to find functions that change the other resources (scrap metal, missiles, drones).

Most of those functions use a pointer to a game structure that holds these values. Scrap and fuel are direct members of this structure while missiles and drones are a little trickier to interpret.

Missiles use an indirect reference using a pointer to another data structure and an offset therein. The drone count is tracked in two places. If you currently don’t have a drone system installed, it’s simply an offset in the game data structure. If you do have the drone system installed, it’s an indirect reference using another structure. The game tracks which one to use with another memory address drone_system_installed which is -1 if no system is installed.

There is a global pointer to the game data structure at 0x8e3500. I found this by just scaning the memory for the address in the heap that I was seeing passed into the functions.

Building the Cheat Engine

Now that I have the data structure figured out and a pointer to it in memory, I can start to figure out a good way to make changes to the running applications. I was originally going to write a C program that used ptrace to attach and modify the structure, but I took a shortcut and wrote a gdb script.

Gdb has a --command flag which lets you pass in a text file that contains gdb commands. It’s not quite a full featured scripting language, but it’s pretty close. Here’s my gdb script for giving me 9999999 scrap and 999 each of fuel, drones, and missiles.

#
# ftl-cheat.gdb
#

# Amount of resources to set
set $scrap    = 9999999
set $fuel     = 999
set $drones   = 999
set $missiles = 999

# Game state address and offsets
set $game_state_addr = 0x8e3500
set $scrap_offset = 0x760
set $drone_offset = 0xbf0
set $fuel_offset = 0x700

set $drone_system_installed_offset_1 = 0x748
set $drone_system_installed_offset_2 = 0x10

set $missile_offset_1 = 0x88
set $missile_offset_2 = 0x2b8

# There are two different locations for drones depending on if a system is installed
set $drone_offset_1 = 0x90
set $drone_offset_2 = 0x290

# Print the game state addr (useful for more debugging)
printf "-------------------------------------------------\n"
printf "Game state address: 0x%08x\n", *$game_state_addr
printf "-------------------------------------------------\n"

# Set resources
set $scrap_addr = *($game_state_addr) + $scrap_offset
printf "Scrap value (0x%08x) is currently:   %d\n", $scrap_addr, *$scrap_addr
set *$scrap_addr = $scrap

set $fuel_addr = *($game_state_addr) + $fuel_offset
printf "Fuel value (0x%08x) is currently:    %d\n", $fuel_addr, *$fuel_addr
set *$fuel_addr = $fuel

set $missiles_addr = *(*$game_state_addr + $missile_offset_1) + $missile_offset_2
printf "Missile value (0x%08x) is currently: %d\n", $missiles_addr, *$missiles_addr
set *$missiles_addr = $missiles

set $drone_system_installed_addr = *(*$game_state_addr + $drone_system_installed_offset_1) + $drone_system_installed_offset_2
if *$drone_system_installed_addr == 0xffffffff
  printf "Drone system NOT installed\n"
  set $drone_addr = *($game_state_addr) + $drone_offset
else
  printf "Drone system installed\n"
  set $drone_addr = *(*($game_state_addr) + $drone_offset_1) + $drone_offset_2
end
printf "Drones value (0x%08x) is currently:  %d\n", $drone_addr, *$drone_addr
set *$drone_addr = $drones

# Quit the debugger
detach
quit

If I combine that with the -p option to attach to a running process using it’s PID, I can attach, run the script that modifies the game data structure, and detach. I wrapped the whole thing in a bash script that makes it even easier.

#!/bin/bash

echo "[*] ftl-cheat"

CURRENT_USER=$(whoami)
if [[ $CURRENT_USER != "root" ]]
then
    echo "[!] Warning: not running as root. May not be able to connect to process"
fi

FTL_PID="$(ps -ef| grep FTL.amd64 | grep -v grep | head -n 1 | awk '{print $2}')"
if [[ $FTL_PID == "" ]]
then
    echo "[!] Could not find PID for FTL.amd64. Is the game running?"
    exit 1
fi

echo "[*] FTL game PID is $FTL_PID"
echo "[*] Attaching to process"
gdb -p $FTL_PID --command=ftl-cheat.gdb -q
echo "[*] Done"

Now all I need to do is start a new game (or continue an existing one) and run

sudo ./ftl-cheat.sh

[*] ftl-cheat
[*] FTL game PID is 16913
[*] Attaching to process
-------------------------------------------------
Game state address: 0x1dee34f0
-------------------------------------------------
Scrap value (0x1dee3c50) is currently:   30
Fuel value (0x1dee3bf0) is currently:    16
Missile value (0x04e48688) is currently: 8
Drone system NOT installed
Drones value (0x1dee40e0) is currently:  2
[Inferior 1 (process 16913) detached]
[*] Done

I removed the lines that gdb dumps on the screen so you can see that it’s attaching, running my script and detaching. After running this, I get far more resources than I can possibly use in a single game. While this should make the game easy, I still had trouble beating the final boss, even with a fully maxed out ship.

FTL: Faster Than Light is a really fun game and I encourage anyone who likes strategy and rogue-like games to purchase it and support the independent developers.

Side Project: Entrance Song

I’ve been working on a side project inspired by a professional wrestling and a blog post I read about a year ago.

I can’t remember the blog’s name, but it described how the local system administrator setup their Sonos speakers to play a theme song whenever an employee entered the office. They used DHCP to detect when an employee’s mobile phone connected to the corporate WiFi and then matched the MAC address to an employee and their chosen entrance song.

The whole idea was pretty amazing and it made me think of my favorite thing about professional wrestling: getting a chance to pick an entrance song for when you enter the ring. It’s something I give a lot of thought to.

This is essentially how I wanted to feel walking in my front door every day after work and also let my friends set their own entrance theme songs.

The implementation used in the blog used tcpdump and some bash scripts, but I wanted to write a version in Python.

Here’s the link to the GitHub repo.

Sniffing the DHCP Transactions

Detecting when someone enters my house is done using DHCP, which is a weird protocol. It’s based on UDP (and implicitly, IP) but it’s used for getting an IP address lease so the device likely doesn’t have an IP address. The solution for this is to just broadcast the DHCP request and response to everyone on the network. The end result is a very noisy transaction, but that makes it easy to sniff.

• A client sends a DHCPDISCOVER packet broadcast to 255.255.255.255
• A nearby DHCP server detects and responds with a DHCPOFFER to 255.255.255.255 containing an IP address
• The client device then sends back a DHCPREQUEST packet (again, to 255.255.255.255) requesting this offered IP address.
• Finally, the DHCP server sends an acknowledgement DHCPACK message, this time just as a unicast packet.

The whole thing is like walking into a hotel lobby and shouting at everyone present until the conceirge hears you and gives you a room key.

For our entrance song to detect a device, we just need to listen for that DHCPREQUEST packet, since it’s being broadcast everywhere.

I’m using Scapy, a very awesome Python library, to sniff my network for DHCP traffic. Scapy is pretty well known for doing cyber security stuff like crafting packets and building sniffers. It’s very easy to write a basic sniffer with a callback.

from scapy.all import Ether, DHCP, sniff

sniff(prn=self.dhcp_monitor_callback, filter='udp and (port 67 or 68)', store=0)

def dhcp_monitor_callback(pkt):
    """Callback for DHCP requests"""
    if not pkt.haslayer(DHCP):
        return
    if pkt[DHCP].options[0][1] != 3:
        return

    mac_addr = pkt[Ether].src
    logging.info('DHCP request from %s', mac_addr)

Here, we’re sniffing for UDP on port 67 and 68 and passing the packet to a callback function. In the callback, we’re making sure it’s actually a DHCP packet and that it’s a DHCPREQUEST packet (the options are a list of tuples of DHCP key-value options. The first tuple is the type of DHCP message, here we’re using type 3 for DHCP requests).

After verifying that we have a DHCP request, we can easily pull the MAC address out of the Ethernet layer. We’ll use this later to compare to my friend’s names in a SQLite database and figure out what song to play.

device = data.get_device_by_mac_addr(mac_addr)
if not device:
    logging.info('This is not a device I know about... Adding it to the database')
    data.insert_device(mac_addr)
    return

if device.owner.song:
    song = device.owner.song
    logging.info('%s is about to enter (%s)! playing %s by %s', \
        device.owner.name,
        device.friendly_name,
        song.title,
        song.artist)
else:
    logging.info('Device owner %s does not have a song. Doing nothing.', device.owner.name)
    return

Meanwhile the SQLAlchemy models are pretty standard classes that represent a Device, an Owner, and a Song and the foreign keys between them. The data.get_device_by_mac_addr and data.insert_device are just shortcut functions for querying and inserting data into my database.

Playing Music

Once we’ve determined who’s entered my front door, we need to play their theme song. I have a Spotify premium account, so I can literally play anything. Spotify provides a pretty good web-based REST API and there’s a Python wrapper for it called Spotipy.

Spotipy hasn’t been updated in a while and the original author doesn’t seem to be merging pull requests. There are a few issues with the library so I forked it and patched them up myself.

Authenticating my premium Spotify account using Spotipy is a little weird. You have to register for their developer program and create a new project. After that, you need to allow the project to access your Spotify account with the appropriate level of access.

For playing the music, we want to do the following:

1. Look up who entered my front door and see if they have an entrance song.
2. If so, save the current Spotify playback so we can resume it later.
3. Fade out the current song.
4. Set the volume high and blast that entrance song for the next 45 seconds or so
5. Fade out the entrance song
6. Fade in the previously played music prior to the entrance and return everything back to normal

Sounds easy enough. There are a few interesting problems to solve though. Spotify doesn’t have a way to play just a section of music and then stop. We can fake this out by starting the music in a new thread, sleeping in that thread for the entrance duration (usually about 45 seconds to a minute) and then stop the music right before ending the thread.

class MusicThread(Thread):
    """A thread to start music and sleep. This is a cheap way to implement playing
    a duration of a song since the Spotify API doesn't include that.
    """
    def __init__(self, sp_context, mp_context, uri, position_ms, duration=45,
                 device_id=None):
        """Constructor
        Args
            sp_context - The spotify context
            mp_context - music player context
            uri - The URI to play
            position_ms - The position in ms to start from
            duration - The duration to play, in seconds
            device_id - The device to play on, or None to play on the default device
        """
        super().__init__()

        self.sp = sp_context
        self.mp = mp_context
        self.uri = uri
        self.position_ms = position_ms
        self.duration = duration
        self.device_id = device_id

    def run(self):
        """Runs the thread.
        Starts playback on the track, sleeps, and then fades out the track.
        """
        logging.info('Starting playback in new thread')
        try:
            self.sp.pause_playback()
        except SpotifyException as e:
            # This often happens if there is no current active device. We'll assume there's
            # device_id being used. The next try/catch block will handle it if not.
            pass
        try:
            self.sp.start_playback(device_id=self.device_id, uris=[self.uri],
                                   position_ms=self.position_ms)
        except SpotifyException as e:
            logging.error(e)
            return
        self.mp.set_volume(self.mp.default_volume)

        logging.info('Putting the thread to sleep for %s seconds', self.duration)
        sleep(self.duration)
        logging.info('Stopping playback')

        # Get the currently playing tack to be sure we're stopping this track and not
        # someone else's.
        current_track = self.sp.currently_playing()
        try:
            uri = current_track['item']['uri']
            if uri == self.uri:
                self.mp.fade_out()
                self.sp.pause_playback()
            else:
                logging.info('Attempted to stop song %s but it\'s not playing', self.uri)
        finally:
            return

Another problem is figuring out how to handle if someone shows up while another person’s entrance song is already playing. Should the previous person’s music be cut off? That’s disrespectful. Also the threads might start fighting over the music controls and we’d get into a weird state.

I thought about writing some state machine code to keep the threads in sync but there’s a much more elegant solution: using Python’s Queue class. These things deserve way more credit than they get. They allow you to have a thread safe way to manipulate a collection. In my case I want to add things without blocking (i.e. someone walks in the door so I need to queue up their song) while removing things with blocking (i.e. making Spotify play the song but block until the song is complete before starting the next one). No awful state machine required!

Here’s the main loop for the music player:

def player_main(self):
    logging.info('Starting music player')
    while True:
        uri, start_minute, start_second, duration = self.song_queue.get(True)
        logging.info('Found a song on the queue!')
        logging.info('Playing %s at %d:%d duration %d', uri, start_minute, start_second,
                     duration)
        self.save_current_playback()
        t = self._play_song(uri, start_minute, start_second, duration)
        logging.info('Waiting for song to end...')
        t.join()
        self.restore_playback()
        logging.info('Song over... waiting for the next song on the queue')

The get method will block the thread until it sees something on the queue. The join method on the MusicThread will also block the main thread until the song completes. The MusicPlayer itself is a Thread so the whole thing doesn’t block the rest of my application while calling these methods.

My Scapy DHCP sniffing code just needs to drop songs on the queue and they will get picked up by the music player thread.

Results and Conclusion

It’s honestly pretty awesome to walk into your own house and have an entrance song play. I now get pumped up as I enter my door and start waving my hands to an imaginary crowd. I’m not sure what my neighbors think of this.

Other Things

• I use some NetGear wifi extenders in my house. It turns out, they change a device’s MAC address by swapping the first 3 bytes to 02:0f:b5. To get around this, I added a --virtualmac option that attempts to fall back to looking in the database for just the last 3 bytes of a MAC address.
• I recommend using a static IP on the device hosting the Python code. I put this code on a Pine64 board I had sitting around and plugged it directly into my wifi router’s wired ethernet switch. There’s no reason it couldn’t have just run on a regular desktop or laptop, but I needed an excuse to use the Pine64.
• It works but I’ve noticed a few bugs, especially when switching playback between output devices. I might get around to fixing them.

Future Work

• I also own some Philips Hue lights and they are a lot of fun. I want to bring those into the entrance by having them dim as the music fades and then bringing them up at a certain point in the music.

Appendix: What my friends picked as their entrance theme songs

• Me - Still Fly (cover) by The Devil Wears Prada
• Alex - So Fresh, So Clean by OutKast
• Yaro (@hokietron) - Bumble Bee Tuna Song by Mephiskapheles
• Samantha - Hide and Seek by Imogen Heap (but at the 2’52” mark where all the memes happen)
• Rob - Imperial March by John Williams
• Suzie (@California_Suz) - Cotton Eye Joe by Rednex
• Smitty - Mah Nà Mah Nà by Piero Umilian
• Patrick - Holy Diver by Dio
• Default song for unknown people - Frolic by Luciano Michelini (better known as the “Curb Your Enthusiasm” theme)

Switching to i3

A co-worker convinced me to try out the i3 tiling window manager. I initially wrote this off as a show of nerdy one-upmanship (I admitted to him I still run Unity on my main desktop) but I told him I’d try it out. After a few days, I have some thoughts in place.

If you’ve never used a tiling window manager, the big difference is that your windows are no longer floating and overlaping. Instead they are mutually exclusive units on the screen, usually arranged in a grid. You lose the traditional desktop metaphor, but you gain some performance and efficiency improvements.

Initial things

I installed i3 via apt-get and started a new session. There was a wave of mild paralysis when I realized I had no clue how to make anything work. Using the windows & desktop model for so long made it hard for me to even conceptualize anything else. I’m used to dragging windows around, maximizing them, minimizing them, sliding them around like physical objects at my whim.

Now I was staring at rigid compartments of cold unfeeling terminal windows with no friendly context menus or hints of what to do next. After learning a few key shortcuts and dmenu, I began to breath a sigh of relief. I could pop open my traditional desktop apps and get my footing. It felt weird to open Firefox and not be able to drag the window around. Even worse, it was frightening when I realized that my usual methods of moving files via drag-and-drop was no longer an option. Opening Nautilus was no longer an option (although if you try, you’ll just get a messed up desktop).

After a few hours of tweaking and seeing what’s available, I had a really attractive clean desktop.

Things I like

For starters, this thing is fast. I can go from login screen to desktop in less than a second. Granted, most Linux window managers tend to be fast compared to Windows, but this is easily the fastest one I’ve personally used.

If you do most of your work in the terminal, then this is a fantastic experience. You can open terminals, have them cleanly arranged on your screen and never need to go searching for the last one you had open among a bunch of minimized windows.

The concept of workspaces is something I never really appreciated until I began to use them in i3. I never used the Unity workspaces because it was easier to just do all your work in one and just minimize the stuff you weren’t dealing with at the moment. In i3, workspaces are a necessity since you can’t just minimize things to a taskbar. The number of workspaces grows as you need them and they quietly disappear after you’re done. This is a fantastic feature I don’t need to dig through empty workspaces trying to find the one I was using. My current setup involves opening my background applications in their own workspaces so I can drop in on them periodically without having them take up my full attention (things like Thunderbird and Spotify). It keeps everything clean and mentally tidy.

Easily my favorite feature is the i3 status bar. It’s such a small part of the i3 experience, but it’s a lot of fun to tweak. Like the rest of i3, it’s simple and aesthetically pleasing.

I fully expected that i3 would take a hard stance on using the keyboard for everything, but I was pleasantly surprised to see the mouse as a fully capable tool. I still use it for switching workspaces.

For my (relatively simple) setup, all my tweaks use only 3 files: ~/.config/i3/config, ~/.i3status.conf, and ~/.Xdefaults. I also like that you can re-load your entire session with a few keystrokes so you can quickly play around with settings and see the results.

Things I’m not liking (yet)

Copy/pasting between windows feels weird, especially if one window supports traditional Ctrl-C/Ctrl-V but the other doesn’t. I don’t think this is an i3 issue but rather something that just isn’t configured in my current urxvt setup.

When doing front-end development, I felt slowed down, mostly because I didn’t have an easy way to view image thumbnails.

Conclusions

I like the change so far and I’m planning on sticking with i3. I haven’t had any compatibility issues with my GUI applications (even Steam runs fine). I haven’t been able to find anything that I absolutely hate.

Since I’m a command-line person, the learning curve hasn’t been too difficult. When combined with a good mechanical keyboard, i3 is one of the most satisfying ways to interact with your Unix-like system.

My setup

Getting it

sudo apt-get install i3 dmenu rxvt-unicode-256color feh

Setting urxvt to be the default terminal

Swap out the command to start a terminal to use urxvt command in ~/.config/i3/config

# start a terminal
# bindsym $mod+Return exec i3-sensible-terminal
bindsym $mod+Return exec urxvt

Get yourself a cool wallpaper

I’m using this one. Add to the end of the ~/.config/i3/config file

exec --no-startup-id feh --bg-scale ~/Downloads/pexels-photo-14676.png

Colors!

Create an .Xdefaults file. I’m using the Railscast theme exported from terminal.sexy along with some tweaks to make urxvt transparent.

urxvt.font:           xft: Dejavu Sans Mono:autohint=true:antialias=true:size=9
urxvt.background:     black
urxvt.foreground:     white
urxvt.scrollBar:      false
urxvt.tintColor:      white
urxvt.fading:         15
urxvt.fadeColor:      black
urxvt.shading:        25
urxvt.inheritPixmap:  true
urxvt.pointerColor:   black
urxvt.pointerColor2:  white

! special
*.foreground:   #e6e1dc
*.background:   #2b2b2b
*.cursorColor:  #e6e1dc

! black
*.color0:       #2b2b2b
*.color8:       #5a647e

! red
*.color1:       #da4939
*.color9:       #da4939

! green
*.color2:       #a5c261
*.color10:      #a5c261

! yellow
*.color3:       #ffc66d
*.color11:      #ffc66d

! blue
*.color4:       #6d9cbe
*.color12:      #6d9cbe

! magenta
*.color5:       #b6b3eb
*.color13:      #b6b3eb

! cyan
*.color6:       #519f50
*.color14:      #519f50

! white
*.color7:       #e6e1dc
*.color15:      #f9f7f3

Tweak the status bar

Copy the existing status bar file and make a few changes.

cp /etc/i3status.conf ~/.i3status.conf

# i3status configuration file.
# see "man i3status" for documentation.

# It is important that this file is edited as UTF-8.
# The following line should contain a sharp s:
# ß
# If the above line is not correctly displayed, fix your editor first!

general {
        colors = true
        interval = 5
        color_bad = '#da4939'
        color_good = '#a5c261'
}

order += "disk /"
order += "run_watch DHCP"
order += "wireless _first_"
order += "ethernet _first_"
order += "cpu_usage"
order += "load"
order += "cpu_temperature 0"
order += "tztime local"

wireless _first_ {
        format_up = "Wireless: (%quality at %essid) %ip"
        format_down = "Wireless: down"
}

ethernet _first_ {
        # if you use %speed, i3status requires root privileges
        format_up = "Ethernet: %ip (%speed)"
        format_down = "Ethernet: down"
}

run_watch DHCP {
        pidfile = "/var/run/dhclient*.pid"
}

tztime local {
        format = "%A, %B %e %l:%M %P"
}

load {
        format = "CPU load: %1min"
}

disk "/" {
        format = "%avail"
}

cpu_temperature 0 {
        format = "Temp: %degrees °C"
        path = "/sys/class/thermal/thermal_zone0/temp"
}

cpu_usage {
        format = "CPU usage: %usage"
}