New Project: Hacknowledge.io

I’ve been quietly working on a new project that just recently went into a closed beta release.

Back in college, I was studying a lot of information security. I was even on my university’s cyber defense team where we competed at a regional competition to defend a computer network against a nasty group of professional penetration testers.

Imgur

I’m in the lower right with those sweet sideburns.

After college, I kind of moved away from doing infosec and I began to settle into a comfortable web application development career. I still tried to keep up on the world of infosec by reading blogs and following news articles, but it’s pretty tough unless you’re actively involved in the community.

Around 2013, I found some co-workers who also wanted to play around with infosec stuff and they let me hop on their private lab to solve challenges, learn some new stuff, and just have some fun. I was hooked back in and I instantly knew I wanted to host my own lab environment to do the same thing. I also wanted to apply some of my software engineer and dev ops skills to try to automate a lot of the boring stuff so I could spend more time exploring security concepts and less time doing admin work.

So that’s why I started hacknowledge.io. It is a private network that I’m hosting and also a set of tools for administering and monitoring the network.

Here’s how it works: I have a server in my basement running VMWare ESXi and with a few really vulnerable virtual machines. All of these VMs are on a private network that can only be accessed by going through a VPN. I’ve been calling this private network the Threat Net even though, at the moment, none of the hosts are dangerous on their own.

For each vulnerable host, there are a few challenge questions. Answering these gets you points and let you move up the leaderboard. The questions are designed to help push you in the right direction of fully exploiting the machine. They start off relatively easy (e.g. Find the port that a web server is running on) and eventually ramp up (e.g. What is the root password?).

Imgur

Currently there are only 3 hosts. Eventually I’m going to have 5 for the initial closed beta round. The first three are relatively easy to exploit. Spoiler alert: the first host is vulnerable to Shellshock, which is about as obvious as they come. Anyone who knows how to install and run a Nessus scan can figure it out. I’m hoping that I can eventually build out some more difficult challenges in the future, but the closed beta is mostly just a proof-of-concept.

In addition to vulnerable hosts, the threat net also has some bots running around doing housekeeping chores like reverting the VMs to a known state every half hour (so you can trash a machine and it will be back to normal so someone else can have a chance). There’s also a bot that just checks to make sure the machines are online and available.

Outside of the Threat Net is a set of hosts that provide a web interface for users to answer challenge questions.

Imgur

In the process of designing and writing all of this, I learned a bunch of stuff. I have a ton of blog entries planned on how I built this thing.

The Ways I've Changed as a Programmer

I’ve been doing professional software development for nearly eight years. That’s not an exceptionally long time, but it’s twice as long as the time I spent in college as an undergrad and it seems like a good place to stop and do some reflection.

I recently noticed at work that I’m now on the top of the heap with more junior developers working alongside of me than ever before. My input is weighed more heavily, my contributions are valued more highly, and I think I’m even getting small glimpses of this new feeling called “respect”. It’s an exciting time.

With all this new clout, I’ve compiled a short list of the ways that I’ve grown as a software developer.

I’ve learned to not be a control freak

Early in my career, I worked on really small teams on small, tight projects. I could know every detail of the baseline and know exactly how all the pieces fit together. I saw the projects as direct extensions of my worth and I found myself really nervous if I heard someone was trying to make some big changes. What if they don’t get my elegant designs and just kludge the whole thing up? What if they don’t have my same skills as me and turn the whole thing into an unmaintainable mess for me to deal with?

I had two options: I could constantly be checking the work of these outsiders and spend hours trying to “clean up” their messy less-than-perfect code. Or, I could just let go of my control freak nature and let them contribute like a professional. This has been incredibly difficult for me to do, but I’m finally getting around to it. If I tried to monitor every single code check-in and every change being made, I wouldn’t have any time to get real work done. On top of that, I’d look like a jerk who can’t play nice with others.

In the end, I really have to remind myself a few things:

  • The success or failure of a project doesn’t depend solely on me.
  • I was once a wide-eyed junior programmer who couldn’t wait to start contributing, no matter how little I knew of the code. I turned out okay.
  • You gotta let things go sometimes. Walking away isn’t a sign of defeat or apathy.

Imgur

I’m less concerned with style

Spaces versus tabs. Semicolons versus semicolon-free. Single quotes versus double quotes. So many battles rage on within software development teams. I have strong opinions of what good clean, readable code should look like, but these days, I’m much less apt to try to enforce it on others. We have a style guide at work for our code and I only like about half of the style choices, but I’m not about to spend the effort to try to change people’s minds on really petty stuff.

I have a theory on this: in college, I was used to doing work and getting a grade on it. It felt good to get validation. In the real world, I might go weeks before getting feedback from someone on my work, and even then, it might just be superficial. For a while, I really craved the kind of validation that I got from jshint and pylint and whatnot. To me, it was a way to write code and have someone tell me that it’s “good”.

I’ve discovered that even well-styled code can still be pretty lousy and I think I’ve outgrown the need for the praise.

My unit testing is way better

Unit testing is a skill that, unfortunately, doesn’t seem to be taught in college and most developers aren’t actively seeking out tutorials or books on how to write good unit tests. Most new developers write really, really awful unit tests, myself included. I could probably write a whole blog post on how you should go about learning how to write a good test suite (I probably will at some point).

In short, your unit tests need to meet some basic criteria:

  • Independent, no external requirements, –that includes databases, network connections, filesystems, etc.
  • Repeatable, you should get the same result each time you run the tests no matter what order you run them.
  • Concise, you should be testing one thing in each test case.

One obstacle for me was learning how to mock out objects and modules for your unit tests. Most unit testing frameworks don’t present this up front and it’s incredibly difficult to write good unit tests without a good mocking framework in your toolbelt.

The Zen of Python is everywhere

The Zen of Python is a short mantra that describes the ethos of the Python community. It’s intended for Pythonistas, but honestly, it works for every language.

When combined with some of the things I’ve gained from functional programming, I’ve found that my code feels much more elegant and even fun to write.

I’m no longer afraid of exceptions

Somewhere back in college, a lot of programmers make the connection that exceptions crash your programs. Therefore, exceptions must be bad and you should do everything you can to remove them from your code.

But exceptions are a very powerful construct! In C code, you have to return positive or negative return codes and the calling function has to interpret them and react. I don’t want to go back to that.

A lot of rookie programmers will eat exceptions and this is universally considered bad form. Another anti-pattern I often see is avoiding situations where explicitly throwing an exception would be appropriate. Usually they will return a null value and let the calling function do a null check to figure that out.

If you’re a junior programmer and want to impress your seniors, learn some of the exception handling idioms of your language and find places where they help your code.

Frameworks are cool, but not a substitute for fundamentals

Frameworks are designed, in part, to reduce boilerplate code and the amount of cruft that you might otherwise write. I used to think that knowing a framework was a fast track for writing better code, e.g. Angular.js has modules and dependency injection so it’s halfway there to making you a JS ninja.

Well, not quite. It turns out, you’ll bend the framework to your bad habits. I’ve seen it so many times that I’m starting to understand the no-framework camp of developers.

For this, I think it’s just a lack of solid hands-on time with a language. I see developers mature in the language over time as they write code and find themselves maintaining it.

I’m going to set a reminder for myself to revisit this in 5 years and see what else I’ve learned. I could have a new job by then, a new favorite language, or a new database that I love. I don’t expect to stay the same, and that’s a good thing.

Learning Haskell

I am spending some time learning Haskell. This is partly because I want to learn about functional programming, but also because Haskell has this aura of hipster programming that I kind of enjoy.

When your industry is growing at a fast pace, it’s a good idea to stay on the edge of things.

Imgur

I’ve attempted to collect together a bunch of Haskell and post them here for my own benefit, but also you can join with me as we throw off the shackles of OOP and welcome our brave new world of functional development.

Intros and Motivations

Functional Programming should be your #1 priority for 2015 by Ju Gonçalves.

A very good brief introduction to what functional programming is. This article argues that functional programming is the next big thing in computing thanks to distributed architectures (better known as The Cloud). Obviously, this article was written nearly a year ago, hence the title. The collapse of OOP didn’t occur in 2015 and it probably won’t happen anytime soon. Still, this was one of the first articles I came across that motivated me to take functional programming seriously.

This quote sums up the whole article nicely:

That promised time when we’d have applications running distributed and concurrently finally has come. Unfortunately, we are not ready: our “current” (i.e., most used) model for concurrency and parallelism, even though might solve the problem, it adds a lot of complexity.

The Dao of Immutability by Eric Elliott.

A fun meditation on the simplicity of functional programming.

All Evidence Points to OOP Being Bullshit by John Barker

Another cathartic rant on object oriented programming. There seems to be a common theme here…

10 Interview Questions every JavaScript Developer Should Know by Eric Elliott.

Wait, a JavaScript article? In your post about Haskell? Whaaaat? Really this is a functional programming rant in disguise. We all know that JavaScript has a bunch of cruft, but in this article, Elliot argues that one of the most damaging aspects of the language is the class.

I advise people to hire based on whether or not a developer believes in class inheritance. Why? Because people who love it are obstinately stubborn about it. They will go to their graves clutching to it … I won’t hire them. You shouldn’t, either. I’ve seen classes wreak havoc on projects, companies, and lives.

According to the author, that’s not hyperbole. Elliott advocates that the best way to develop JavaScript is to adopt some of the functional traits of langues like Haskell. Being a full-time JavaScript developer, this was the first article that caused me to sit up and take note of functional programming.

Why Haskell Matters from the Haskell.org Wiki

Although it’s fairly dense, this is reasoning for Haskell from the people behind it.

Books

Unless otherwise noted, all these books are freely available online.

Learn You a Haskell for Great Good! by Miran Lipovača

This is the definitive beginner’s guide to Haskell. I remember picking this book up at a Barnes and Noble and thinking what the hell is going on with this language? Years later I actually read the book and I enjoyed the nice pace and quirky illustrations.

Real World Haskell by Bryan O’Sullivan, Don Stewart, and John Goerzen

Haskell is often unfairly categorized as an “academic” language that should remain confined within the realm of computer science theory. This book is both a tutorial and counter-argument to prove Haskell can be used in software development for serious development.

The Structure and Interpretation of Computer Programs by Harold Abelson and Gerald Jay Sussman with Julie Sussman

To be honest, I haven’t read this text. Also, it’s not actually about Haskell, but rather the overall concept of programming language design. It also makes use of Lisp instead of Haskell. I’m only including it here because it’s a monumental work that is often recommended when learning about functional programming.

Videos

Make Haste: Fast Track to Functional Thinking by Katie Miller at CampJS

This is a very fun talk given at a JavaScript conference. It includes a brief overview of functional programming, Haskell, and a framework called Haste used for building web applications in 100% Haskell.

Interesting Projects

Purescript

A Haskell-inspired language that compiles to JavaScript. The language itself is implemented in Haskell, of course.

Haxl

A Facebook open source project for simplifying remote data access.

PostgREST

Arguably the most popular Haskell project around for now. This is a web application that gives your PostgreSQL database a REST API. It’s written in Haskell. I find this to be an interesting project since it is something that most people wouldn’t consider using Haskell for.

Hakyll

Everyone knows about Jekyll, the Ruby static site generator. Hakyll is a Haskell implemented static site generator.

Geospatial Resources

My day job involves doing a lot of GIS work (that’s Geospatial Information Science, for you non-map nerds). Along the way, I’ve collected a lot of interesting data sources for building maps.

I’ve decided to organize them all here. I believe that maps tell a story and by providing tools to build good maps, we can tell better stories.

Some of these links may be broken in the future or the websites undermaintained. If you want to submit a website, get in contact with me.

Data.gov

The U.S. Government has a ton of geospatial data. The biggest challenge is finding it. Fortunately, the recent development of Data.gov has made it slightly easier. Data.gov is supposed to be the one-stop location for all public data, however there are still a lot of individual repositories of data out there that aren’t feeding into it.

Be sure you select the geospatial tag when filtering search results.

Links:

NASA

You already know who NASA is. They operate a very nice website called NEO (for NASA Earth Observations). In particular, the true color 1-day imagery is pretty amazing. They also run a separate website called EarthData which has even more imagery.

Imgur VIIRS 1-day for July 17, 2015

Links:

US Department of State

A lot of the data from individual branches of the federal government gets syndicated to Data.gov, but often they get lots in the mix. The Department of State now publishes to the State GeoNode server.

Imgur Syria Internally Displaced Person (IDP) Sites, April 16, 2015.

Links:

United States Geological Survey

The United States Geological Survey also warehouses a ton of data and makes it available through their Earth Resources Observation and Science (EROS) webpage. It requires registration to download the data and I’m not sure if it gets syndicated to Data.gov.

Links:

U.S. State Governments

Ideally, all state GIS data should be going to Data.gov, but it can usually be easier to get the data directly from the state government’s website. MIT has collected the web site for every single state GIS group to help you cut down on your searches.

Links:

National States Geographic Information Council

While they have a boring name, the NSGIC has a really cool database called the GIS Inventory.

Links:

United Arab Emirates

The UAE has spent a lot of effort to build out geospatial collections of the country, which results in some breathtaking imagery of this nation. I suppose if you’re building the world’s tallest skyscrapers and creating artificial islands, you want to show them off.

The Abu Dhabi Geospatial Portal is a good starting point to explore some of this data. Despite the name, it has a lot of imagery for all of the UAE including Dubai.

Imgur Ferrari World at Abu Dhabi, UAE

Links:

OpenTopography

Funded by the National Science Foundation, OpenTopograhy is a website that allows you to search through a lot of topography data. They have a very nice interface for finding terrain data and then processing it into one download.

Links:

DigitalGlobe

Being a commercial company, DigitalGlobe doesn’t just give away all their imagery. They do have a few good samples of large cities, plus a gallery of their work.

Links:

NOAA

The National Oceanic and Atmospheric Administration (NOAA) is a government agency that deals with weather, the ocean, and the earth in general. It’s not surprising that they have a good amount of free GIS sources. One of their nicer sources is nowCOAST, a group of services for providing near real-time observations and weather watches and warnings. If you’re a weather nerd, this is some awesome data. They provide a map viewer, but the raw feeds are more interesting.

Imgur NOAA Near Real-Time Observations service weather mosaic overlaid on the Mississippi River

Links:

A Geospatial View of the United Arab Emirates

I do a lot of work with geospatial data at my day job. Occasionally, I’ll find a cool data set and I’ll save it off so I can spend some time looking at it and seeing if I can find anything particularly interesting. I found an ArcGIS server hosted by the United Arab Emirates that hosts some very nice imagery of the country.

The imagery is part of the Abu Dhabi Geospatial Portal. Despite the name, the project actually has imagery from all over the UAE, including Dubai.

According to the metadata, this imagery comes from WorldView-2. You can click any of the images to get the full size version.

Here’s a wide shot of the city of Abu Dhabi. In case you’re wondering, Abu Dhabi is not the same place as Dubai. It’s southwest of Dubai and is the country’s second biggest city and capital. This is a beautiful shot of the city meeting the Persian Gulf. But wait… what’s that red triangle in the middle?

Imgur

Zoom in!

Imgur

Is that a Ferarri logo?

Imgur

This is Ferrari World Abu Dhabi. The world’s largest indoor theme park and currently the only Ferrari theme park. There appears to be a racetrack nearby for visitors to test drive a high-end sports car.

Imgur

Moving to Dubai in the northeast, we can see the famous man-made island archipelago, The World. It doesn’t seem to be doing very well. In addition to being underdeveloped, the islands appear to be sinking into the ocean.

Imgur

At least one person seems to have made an investment in a private island in the North America section.

Imgur

There are no roads connecting this to the mainland. The only way on or off this island is by boat or helicopter (note the helipad on the western part of the island). I guess if you like privacy, then this is worth it.

Imgur

The Palm Jumeirah appears to be faring much better. This is one of two palm-shaped island formations (the other being the Palm Jebel Ali). A third one was planned, but has not been developed.

Imgur

A detailed look at the spine of the Palm. It’s mostly developed, although you can see one of the fronds is still empty.

Imgur

The most expensive houses are on the tips of the fronds. This one, on the eastern half, is particularly impressive. While the outer ring of islands is made up of hotels, the inner fronds are private residences.

Imgur

The canal work in downtown Dubai is impressive. This image also has the most famous landmark in the entire UAE…

Imgur

A more detailed look at the Burj Khalifa, the tallest man-made structure in the world.

Imgur

I honestly don’t know what this is; some kind of racetrack perhaps. It has smaller concentric tracks within it.

Imgur

Some agriculture imagery. As you can image, there’s not a lot of this in the desert. Center pivot irrigation, the style of farming here, makes for those circles which are fun to look at.

Imgur

A very wide image of a highway outside of Dubai. I like this because it shows they very carefully planted trees around the edges of the highway. It’s probably to keep sand from blowing on the road, but I’m guessing it also creates an illusion that you’re not actually driving through the desert.

Imgur

When planning neighborhoods and infrastructure, these UAE engineers really like geometric patterns, particularly concentric circles.

Imgur

More ornate geometric patterns, this time in the form of a highway interchange.

Imgur

I don’t know what this is. It appears to be some circular roads with some well placed trees extending outwards, but I can’t tell if this is supposed to be a neighborhood development. I think the engineers are just messing with me at this point.

Imgur

This looks like some kind of resort hotel.

Imgur

These are some very big yachts. They don’t appear very large until you compare them to the size of the cars parked nearby.

Imgur

I would occasionally find these compounds that are fenced off. I didn’t look up what they were.

Imgur

There was a lot of other stuff in the data set. I might return to it and try to find some more crazy stuff. If you’re into this kind of thing, then you should consider learning about geographic information systems (GIS).

Imagery Credit: Abu Dhabi Geospatial Portal