Simplify Your Code: Functional Core, Imperative Shell

11 points by hwayne

zmitchell

Similar topic from Gary Bernhardt:

quad

"Similar" 🤣 AFAIK he coined the phrase and his talk is linked at the bottom of the post!

Learn more in Gary Bernhardt’s original talk about functional core, imperative shell.

The choice of example involving a database is an interesting one for sure. If I were looking at that code, I would be asking things like:

Why is filtering not pushed down into the DB query that is getting the users?
Is it possible for there to be a lot of users/what's the bound on the memory usage?
Where is the error handling logic for email sending (potentially partially) failing? Where do retries happen?

Now, of course I understand that it's meant to be a small toy example. But I find that trying to apply the idea of functional core/imperative shell for this kind of DB-related code basically ends up forgoing proper of database query planning optimizations, transactions etc. in favor of trying to handle everything in the application code.

aiono

THIS. I really like the idea but for DB-heavy applications I think it falls short quickly for the reasons you stated.

xavdid

I find that I end up writing a lot of CLIs this way. There's a bunch of pure functions that are easily testable and then the top layer of CLI commands that call into those functions, but don't have much test coverage directly.

In practice, this works pretty well. the CLI commands don't change all that much and compiler / typechecker verifies that the CLI is supplying arguments for everything in the function, so drift is easy to catch. It works nicely!

ph14nix

I've seen people do a functional core even when the core logic is inherently interactive. The core logic gets implemented as a state machine f :: State -> InEvent -> (OutEvent, State). Your OutEvent could be an outside call you make which will deliver its result in the next InEvent.

It's surprisingly not bad when you're ok with some performance hit, and when your language supports concise ways of expressing such f. For example, in Haskell it's trivial to hand-make a monad (4 lines of code, roughly) where you can write the f as a regular sequential algorithm to be automatically translated to a state machine. Or if your language supports coroutines, e.g. Python generators, then you can abuse those to write state machines in sequential style.

def f():
  name = input('name: ')
  print(f'Hello {name}')

So something like the above is broken down into this.

def f_core():
  name = yield { 'func': 'input', 'arg': 'name:' }
  yield { 'func': 'print', 'arg': f'Hello {name}' } 

def f_shell():
  coroutine = f_core()
  req = coroutine.send(None)
  while True:
    try:
      if req['func'] == 'print':
        res = print(req['arg'])
      elif req['func'] == 'input':
        res = input(req['arg'])
      else:
        raise "Unknown function"
      req = coroutine.send(res)
    except StopIteration:
      break;

Maybe not best Python style, and clumsy without types, but it's very natural if you're used to purely functional approach. And it's is much cleaner if you're writing it in a functional language.

It gets terrible very quickly though when you're writing something like old C++ with no coroutines, having to unfold sequential code with loops and states all over into an explicit state machine is a burden hard to justify. It's unclear whether any benefit of the functional core is worth it. I guess, there's a limit to how much you can achieve with a good functional idea in a poor language.

quad
It's unclear whether any benefit of the functional core is worth it. I guess, there's a limit to how much you can achieve with a good functional idea in a poor language.

You'd be surprised. The State -> Event -> (Effect, State') signature shows up in a lot of places that don't have coroutines:
- Erlang's gen_statem and even gen_server at the handle_* level
- The Elm Architecture was distilled from FRP and later influenced Redux
- The write-ahead logs of almost every ACID database
- Event sourcing as found in many high scale backend services
- Immediate mode graphics, where Effect is a command-list to render
- More deeply, these are all essentially different takes on a functional event loop:
```
state = {}
event = Start
while event:
    state, effect = App.update(state, event)
    event = IO.do(effect)
```
The huge benefit is that a functional core is trivially validated. Or, in the infamous words of Joe Armstrong:

You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.
- ph14nix
  
  I'm only superficially familiar with Elm, but quick googling tells me it lets you define Monads with do-notation. This is lets you make your own EDSL for coroutines, and tailor it even better to your own use case than vanilla coroutines would be, doesn't it?
  
  By validating functional core you mean testing it? My complaint about languages with no coroutines was that if the logic in your functional core is inherently sequential and effectful, then unfolding it into an explicit state machine transition function makes it difficult to understand and thus to validate using a code review.
  
  Imagine a sequential algorithm with some fors, ifs, breaks, continues and effectful external calls sprinkled in-between. Assuming this is a natural way to express your domain-specific core business logic, unfolding it into a transition function means you have to keep all your local variables in the state record, keep track of 'program counter' value that tells where you are in the imperative version of the code, and then unfold the loops and conditionals into a goto-style spaghetti in the transition function.
  
  I get that there's plenty of use cases where the logic is not as dramatically sequential, and those tools do great there (my top comment fails to acknowledge it properly), buy they do choke on cases where the logic is sequential.
  - quad
    
    […] but quick googling tells me [Elm] lets you define Monads with do-notation.
    
    No, AFAIK.
    
    My complaint about languages with no coroutines was that if the logic in your functional core is inherently sequential and effectful, then unfolding it into an explicit state machine transition function makes it difficult to understand and thus to validate using a code review.
    
    I've had similar complaints. Then I wrote some scaffolding to do exactly this in a major consumer mobile app. tl;dr coroutines didn't help.
    
    Most event loops involve concurrency. At the bottom, perhaps a set of communicating sequential actors. But, consider even a basic multi-page form:
    
    User can enter their name; the "next" button becomes active when the name field is non-empty
    
    User can enter their birthday; the "back" button is always active and returns to (1); the "submit" button becomes active when the name field is non-empty.
    
    How would you express that with coroutines? IMHO a "back" button in particular sucks without tail-calls.
    
    Worse still, without significant runtime assistance, coroutines practically mandate replay-to-recover. That is, we can serde explicit state and use that to continue processing after a crash or on another machine or on a new version of the application. It's hard to serde a coroutine and very hard to migrate coroutines with structural changes. Most runtimes punt on the problem entirely.
    
    ph14nix
    
    The sequential case I originally had in mind was one where the functional core issues synchronous calls to the outside. Sort of like my example in Python. Yours is more reactive (with a sequential flow the user may follow to reach termination), but let me think of how I'd approach it.
    
    First, I'd imagine what it would look like if I wasn't making a functional core, but could directly block to wait for events with next_event() and issue effects with updateUI(). The language you use in your problem description (in English) is an imperative one with goto. So if I can use something like pseudo-C++, I can directly translate your words into code (no functional core yet) by putting steps one after another in the order of the flow, storing the form data we've seen in the local variables, and handling "back" button with a goto.
    
    Result form() { String name = ""; name_page: while (true) { updateUI(NamePage{ name = name, next_active = (name != "") }); match next_event() { // match is Rust-style NameUpdate { new_name } => name = new_name; NextClicked => break; } } // move declaration up if you want it preserved between pages int birthday = -1; birthday_page: while (true) { updateUI(BirthdayPage{ name = name, birthday = birthday, next_active = (birthday != -1) }); match next_event() { BirthdayUpdate(new_birthday) => birthday = new_birthday; BackClicked => goto name_page; NextClicked => break; } } return Result { name, birthday }; }
    
    Now, to turn it into a functional core with coroutines, syntactically replace each block (the effects are only used in blocks like this)
    
    updateUI(x); match next_event() {
    
    with just match yield(x) {. Would that work? Have I missed anything? I used goto, true, but I feel it was necessary for a natural expression of the algorithm here. Without goto, we get a mess with or without functional core, right?
    
    I guess one point you're making is that some problems are almost equally painful to write with and without functional core, i.e. you're more or less forced to write it as an explicit state machine anyway, so the cost of making a functional core is little. In the example above, if my language didn't have goto, I'd probably resort to explicit state machine in both cases.
    
    I've encountered what you say about serde of coroutines too (Haskell, in my case), and I agree it's a significant limitation. I ended up not using them and forcing an explicit state transition function.
  - schuelermine
    
    I don’t think this is a good principle in general. For instance, you might need to decide what to do next based on the result of the email sending. But those criteria are pure, so should you factor them out? And what if I need to do some extra fetching for some of the users based on a pure criterion?
    
    A more general and lax principle that I would endorse would be to try to mold code into an easy-to-understand shape such as IO → pure computation → IO whenever possible, to reduce mental strain at any given time.
  - BinaryIgor
    
    In general, separation of concerns is a good mindset to have, but:
    
    In most cases, you would push filtering logic of this kind to the database
    
    You still can have bugs at the integration points - the fact that you isolated those pieces of logic, like getExpiredUsers and generateExpiryEmails in the example, doesn't prove that at runtime they're called with proper & expected input values
    
    Mostly for the 2. reason, I'm a big fan on in-between, integration tests. I think testability should be evaluated in the broader context than is presented here
  - mccd
    
    I think this pattern can be overkill for logic-light CRUDs, because generally the part that is most interesting to test is the "full flow". When you test the full flow, you generally don't need as many mock functions, and the mock functions can be reused between tests.