Separation of What from How

I've been trying to write a post on declarative programming for awhile now, but it always felt a bit too academic for my taste. And a bit too academic to be useful, frankly, because if you're a part of a team, you're not going to go guns blazing into your project screaming "we're switching to Haskell!" Or OCaml.

No, there's gotta be a way to both get the right gut feeling about what is or isn't declarative, and be able to sneak it into production code without having to ask for permission.

So I've decided to go over how my style of writing has moved from obnoxiously imperative to more declarative over the years. What helped me get there and how you can do it too.

The beginnings

It all really started for me with an article by John Carmack in 2012, titled "In-Depth: Functional programming in C++" it contained, among others, this gem of a quote:

No matter what language you work in, programming in a functional style provides benefits. You should do it whenever it is convenient, and you should think hard about the decision when it isn't convenient

Reading this, the 24 year old me thought "you have my undivided attention". Or he would've if "Django: Unchained" wasn't almost a year away from the Polish release at that time.

The article then goes in-depth (duh...) into pure functions and how to do functional-esque programming in C++. I highly encourage you to read it, even if you're not into C++.

That article doesn't talk directly about declarative programming, but once you start googling for functional, you're destined to find that as well. Which is how I've stumbled upon the most common definition of "declarative programming":

Focusing on the "what" rather than the "how"

But what does that mean, exactly? Well, I can say I didn't fully understand (or feel like I understand...) that until I've listened to the numerous talks by Kevlin Henney. Once you're finished here, grab some popcorn and run this playlist.

I think that the best way to start really understanding it is to look at the single simplest change you can make to your code, and the way you write it, to bring it closer to the declarative side of things, and thus to arguably improve it.

Disclaimer

I'll be describing this through the prism of object oriented programming, which Wikipedia puts under the "imperative" umbrella. That makes sense historically but not necessarily logically.

My advice is... don't think about this too much. I'll explain the distinction between declarative and imperative in a sec, but what's important is that OOP can be entirely imperative or entirely declarative or anything in between.

I'd also argue that declarative and imperative ends of the spectrum could just as well be labeled "good" and "bad", but I know not everyone agrees. And that's fine, as long as I don't have to work on their code 🤣

Show me the code

Let's look at the simplest possible example, capturing only the essence of the distinction, as well as how it's inescapably intertwined with separation of concerns and encapsulation.

This code is on the imperative side:

def download_new_users(self) -> List[User]:

While this one is more declarative:

def new_users(self) -> Users:

There are a couple of differences here. Most notably, the first example contains a prefix "download", which tells you exactly how it obtains the users - it downloads them, possibly from an external service.

Is this really something the code using that method should care about? I'd argue not. This is an example of an implementation detail leaking outside encapsulation, even if "just" in the name. If you come up with a better (faster, more reliable, etc.) way of obtaining a list of new users, what are you going to do? If you leave it like this, it's going to become misleading. If not, your only choice is to change that name in the entire codebase. Now, I realise that your IDE will do it for you but still... yuck!

Even worse, that method returns a list of User objects. It's better than a list of dictionaries, but it is yet another implementation detail leaking outside. Moreover, it creates adhesion between that method and... literally everything else using it.

Kevlin Henney often says that returning basic types from functions is the ultimate destruction of encapsulation. For instance, if I wanted to further filter that list of users, I'd have to write something like this:

kents = [u for u in download_new_users() if u.first_name == "Kent"]

And I would probably be tempted to do it inline, because hey... I need it once.

Instead of doing this:

kents = new_users().named("Kent")

Now... you may think it makes no difference, because it's highly plausible the actual implementation will look something like this:

@dataclass
class Users:
    _users: Iterable[User]

    def named(self, first_name) -> Users:
        return Users(
            [u for u in self._users if u.first_name == first_name]
        )

So there's no real difference for the computer! To which I say:

Any fool can write code that a computer can understand. Good programmers write code that humans can understand. — Martin Fowler

new_users().named("Kent") is more immediately understandable to a human than the alternative, and it's a hill I'm willing to die on. You could show this piece of code to a complete layman and they'd tell you immediately what this code does because it says it in plain english (extra syntax notwithstanding). Try doing that with a list comprehension.

But there's more to it. It's not only more declarative, it's also better encapsulated, which usually goes hand in hand. Which means I can change that implementation... For instance, to one that uses generators or lazily builds a database query (similar to an ORM). And, most importantly, that would not affect the rest of the code in any way. It would be entirely transparent. My code's behaviour becomes implementation-agnostic!

I know what I just wrote may not be groundbreaking to you... But, judging by all the code bases I've seen within the last decade or so, it may very well be to many.

Domain Specific Languages

One of the coolest uses (if not consequences) of writing in a declarative way is domain specific languages, or DSLs. What is that? The usual example is SQL:

SELECT * FROM users WHERE name="Kent"

It's fairly obvious that this language is domain-specific. You're not going to write a generic program with it, you're just going to select "Kents" from the "users". Do you care exactly how the database will do it? No. You may give it hints, you may ask it to explain, but all in all it's not a for with a nested if.

The thing is, our example of new_users().named("Kent") can also be considered a DSL. The best thing about DSLs? Done right, and coupled with overall domain-driven design, they basically end up looking intelligible for domain experts. It doesn't mean they'll be able to modify the logic, but they may be able to read it, or at least to understand you reading it verbatim (removing just the syntax, which may be confusing to some).

The best thing is, done right a program expressed in a DSL will be dead easy to modify. Not just because it'll be easy to talk about it with domain experts, but because the domain will probably change evolutionarily rather than revolutionarily. If you start with a language that follows the problem domain closely, there's a higher chance it'll be able to keep doing that as the domain evolves.

Naming

All in all, everything comes down to naming. Code is made of code. It's a shapeless, isotropic blob, and the only thing breaking the homogeneity of your program is how you name things and groups of things. And possibly whether you name them.

In fact, one of the single best metrics I can think of when it comes to code quality is the ratio of things named to things unnamed. What do I mean by that?

Well, look at this piece of code for the (insanely great, btw) astpath tool for Python. This file contains 81 meaningful lines of code (disregarding imports) and yet merely 3 functions. In other words, only 3 "building blocks" of that code are named.

Sure, there are variables and imported classes, but unless you really dig into e.g. the block between lines 61 and 80 there's no way in hell you'll understand what it's supposedly doing. You can, however, very clearly see how it does whatever it does. It utilises etree, there's a conditional statement and a for loop with some conditionals thrown in for good measure.

Looking at this code at a glance I can roughly tell how much work the interpreter and the CPU are going to have running it before I understand what it even does. What jumps out at me are the language constructs, not the intent of that particular block.

Uncertainty Principle

And that's the key distinction between imperative and declarative programming. I like to call it the "programming uncertainty principle". The more exposed the implementation (imperative) is, the less obvious the intent (declarative) and vice versa.

The fun part is that an API which looks very declarative from the outside, may very well be extremely imperative on the inside. In fact, sometimes it will have to be for performance reasons, because imperative code will, in many cases, be quicker than declarative code.

The paradox of optimisation

That said starting at the imperative end of the spectrum for the sake of performance is usually a sign of premature optimisation. I've seen people avoid function calls in Python, due to them being relatively expensive in the absence of inline, leading to 1000-line long functions, impossible to read, comprehend, modify, and (amusingly, yet consequently) impossible to optimise.

This is the paradox - while imperative code may very well be quicker, it's still better to start with declarative code, find (through profiling) the exact areas that are slow, and re-implement them in a more imperative style (or in a different language) behind a declarative facade. Doing it the other way around... may prove impossible, especially given that highly imperative code tends also to be less testable.

How to start?

How to start writing more declarative code? First focus on naming. Force yourself to avoid prefixes, including ones like "get" and "set". This will be hard at first.

Then, always think twice before returning a basic type from a function. This will feel like needless extra work at first. Once you're comfortable with that, you may stop using basic types for your fields. For instance:

@dataclass
class CreditCard:
    number: str

becomes:

@dataclass
class CreditCard:
    number: CCNumber

This will feel like needless extra work at first, before you realise it allows you to parse, not validate. From there, nothing will be the same.

Finally, do TDD. And yes, that's the last step. Because while I believe TDD will greatly support you in those efforts, going into it without the right mindset you will likely end up testing implementation details. Which has the potential of scaring you off TDD for good...

Test-driven declarative

Thinking back, I'm fairly sure it was the pursue of declarative programming that brought me to TDD in the first place (along with laziness). Test-driven development is, by its very nature, declarative.

You state the circumstances, you state the end result, and you put an empty shell of a behaviour subroutine in between. I emphasise the "behaviour" part here, because you should be focusing on what the code does rather than what it is.

I’m sorry that I long ago coined the term "objects" for this topic because it gets many people to focus on the lesser idea.

The big idea is "messaging."…The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be. — Alan Kay

Once you have that, you fill that shell with implementation, which is completely disjointed from the test. Then you refactor, which could mean just moving things around, but just as well changing the implementation drastically, while keeping the API unchanged and the outside world unaffected. And then you may also have multiple implementations for different circumstances.

TDD as an early warning system

Eventually you may find your TDD cycle slowing down. Test preparation becomes more tedious, refactoring becomes harder, or implementing one thing requires touching more than one place.

More often than not, this indicates you have a leaking implementation detail somewhere. Which usually means there's some area of your system which is not declarative enough. That's where you should stop wrestling with implementing your feature and fix that problem.

for each desired change, make the change easy (warning: this may be hard), then make the easy change — Kent Beck

The point of software is to change

Programmers often think their job is to make a computer do something, but that's not true.

We've been able to do that long before programming was even invented. Dedicated, non-programmable hardware is more than capable of doing a single thing perfectly forever, and it would probably do it far more efficiently.

The purpose of programming and of a programmer is to make a computer do one thing today, another tomorrow, and yet another the day after. Programming in a declarative way aids separation of concerns, which is great at helping you achieve that at a steady, sustainable pace.