Adrian Schneider

Thoughts & Rants on Software Development

Jan 2014Lossless Expression

Software developers are hired to solve problems with technology. When the solution is known, we are implementing that solution with technology. The choice of technology is often more apparent, and the job is more comparable to following a manual. When the solution is unknown, things can become a little more challenging. Our emphasis should shift from technical issues to the expression of our intent.

Many programming languages, frameworks, databases and other tools, limit the way we can express our intent. We tend to unknowingly favor abstractions, meaningless components, and tools that meet technical problems we aren't even facing yet. We focus on how data is stored, transmitted, and the components that allow us to do it quickly. We favor lossy technology that does not allow us to fully express the problem we are solving. Our intent is lost and becomes difficult to express.

With an emphasis on expression, development can become an iterative exploration of the problem domain. Language, boundaries, objects, actors, processes, transformations and workflows emerge as we try to crystallize our view, and our representation of it.

It's important that you do not let your technology choice dilute your intent. Different programming paradigms will lend to different problem domains. Can you represent the journey from a request to a response through a series of clear transformations? Does each state between those transformations have a name, and clear properties? Can you model the messages between components and the boundaries they cross as objects/namespaces? How about as a long procedural script? Can you stuff a series of interactions into create/read/update/destroy commands? How lossy is it - and is that acceptable?

One added benefit of working this way is low amount of cognitive load it requires. There is always some internal translation or mapping of what you're trying to express, and the code that makes it happen. The less loss that happens here, the easier it is to understand, and the easier it is to change.

If you subscribe to the idea of building your domain layer first, maintaining an effort to express your domain as articulately and losslessly as possible is very powerful. It becomes a natural process of exploration, and separates the programming from the problem solving. Each specific problem you need to solve becomes very isolated, and has a clear connection back to the overarching goal.

As you work your way outward to user interfaces, or various supporting services, define the most expressive and ideal interaction first. If you need to compromise later for technical reasons, that's fine. Contrast with that the opposite: if what you built suffered major losses early on, it's much harder to understand and enhance later.

Focus on the lossless expression of your intent. Good design will follow.

Jan 2014Misplaced Testing Confidence

I've been told that writing tests ensures your software works properly. Unfortunately, no amount of tests will 100% tell you that your software is bug free. It can't. Testing is all about adding confidence. Confidence that your efforts are going towards the right goal.

We tackle logical complexity with unit tests. Tests for every code branch, and ensuring our outputs match our expected outputs for every set of inputs. We test for catchable failures. These tests are lightning fast: they simply build and manipulate data structures to meet some assertion. At this level, we are making sure that this tiny unit does its job exactly as its been programmed to. The job you told it to do. Now, that job may be wrong in the first place, but even so, it's successfully achieving your failure.

Now, that unit may not even be compatible with the rest of the system. So we write integration tests. This ensures that units are compatible with each other, and can successfully be wired together to perform tasks. These tests are simple: connect A and B, and hopefully they yield C. They are a little slower, because they require some form of a running system. Luckly, less of these are needed.

Even with perfect units doing what their told in perfect harmony, you may not be solving business problems yet. Working code is nice (or at least, slightly better than non-working code), but if it's not serving its purpose, is it anything more than dead weight?

Confidence gives us power. The power not to worry, the power to move onto the next task. In other words: feedback. Letting us know when we are done, and when we still have plenty of work to do. Testing creates a feedback loop as we work to tell us if what we built actually works. TDD gives us a very tight feedback loop: failing test, write code, passing test, etc.

Why are so many developers leaving the most critical test until the end? The acceptance tests. Does this code actually solve the problems it was created to? If it doesn't, how much time was wasted writing code before this was realized? That is the feedback loop you need to tighten.

It's easy to work in a silo, and write beautiful code that is "bug free", fully covered, and runs lightning fast. It's easy to merge that into an application and wait for bug reports inevitably come in. You may think you are performing because you're delivery high quality code and getting your job done. And you may be. Or maybe not. You won't know unless you actually test against your requirements.

One way you can ensure you are on the right track is creating two nested feedback loops. The outer loop, testing against business requirements. You can formalize this with tools like Behat or Cucumber. As an example, these may drive use cases and check the results. It will depend on your domain. And that domain will depend on which layer you are working in.

The problem domain of your web layer is HTTP. Your acceptance test suite should, as expressively as possible, assert that various HTTP request messages are responded to with the corresponding HTTP response messages. That's it.

Your business domain layer should, as expressively as possible, assert that your business objects can interact under the correct conditions, with the correct outcomes. With an expressive and accurate ubiquitous language, you can ensure those concepts drill all the way down to the code, rather than just at the periodic status updates.

While you are implementing these acceptance tests, you'll write units of code that integrate with the rest of your system. And you'll write tests to ensure those things do those jobs. When you are all done, your business tests should pass. And you know you'll be done.

With these tests driving your progress, you can be confident that your solution is actually solving the business problems it was meant to. When your code leads you astray, you'll quickly realize that they aren't helping you achieve your goal, and you'll find your way again.

Dec 2013Industry Ideals Disconnect

There's an alarming large disconnect between what our industry preaches as ideals, and what it actually produces. Ultimately, I believe this stems from developers wanting or needing to just get shit done. Working code is seemingly good enough, and anything more is an unknown. So we carry on.

I haven't been quite so lucky. My first project finally shut down last month after being in production for about ten years. My second project still requires the occasional support request, also ten years later. Since those initial projects, I've worked on hundreds of projects of various sizes, some failing, and some slipping past deadlines endlessly.

It's a tough job. There is an infinite number of things to know, and you have to balance all of them against your unique constraints. When we seek help, the resources offer shallow solutions. Tutorials aim to get you started, frameworks aim to accelerate, but nobody can paint you a full picture. We navigate an impossible maze of decision making balancing doing things academically correct, doing things quickly, and actually solving the business problems we were brought on to solve.

Unfortunately, what has happened is that we tend to favor publishing and reading content that shows us how to get shit done. Anything more is silly, and perhaps is reserved for the book authors and consultants who have clearly lost touch with reality and want to take advantage of us. For those who do accept some of these ideals, we cannot always practice them.

Ideals like...

  • Accurately model problem domain
  • Separate responsibilities, exposing clean interfaces
  • Layer applications with clear boundaries
  • Unit test every line of code or every logic branch
  • Business tests for every use case or interaction
  • Version every change cohesively and articulately
  • Support multiple communication channels (HTTP, CLI, etc.)
  • Communicate over those channels in various formats (JSON, XML, etc.)
  • Minimize cost of utilization (battery, memory, disk, bandwidth, etc.)
  • Support interaction in different locales
  • Protect users and their data from harm
  • Be simple enough to allow adoption and educate users
  • (searching things programmers should know should yield millions more)

Gaps like...

  • The vast majority of developers do not write tests.
  • The vast majority of developers lean on frameworks for design guidance.
  • Building applications from the inside out and testing the logic that matters first.
  • Many of the above ideals only come up when they become problematic.
  • Interacting with RESTful web services that promote discoverability. No popular frameworks or clients that I've seen can do this.
  • Not relying on poorly designed ORMs when persistence isn't a concern yet, nor is SQL a problem until much later on.

Let's start asking how we can do things better, rather than how we can do things faster. For 2014, my goal is to dive deeper into these topics, and to try and be more mindful of what I produce. Every snippet, article or piece of code I publish will be focused on raising awareness, rather than a cheap trick to save a buck.

These things can't be afterthoughts.