Hof Pattern

view on wiki.c2.com
last modified: December 23, 2013

This is an example derived from discussion in ArrayDeletionExample, intended to show a generic HOF UseCase.

Imagine we have a complex, difficult-to-partition algorithm or process -- let's call it complexCustomisableAlgorithm (see below) -- which requires some context-dependent customisation. The canonical example is a sort algorithm -- the comparison function needs to be customised depending on the data types we need to sort -- but the same approach is applicable to a variety of situations.

In a business context, complexCustomisableAlgorithm might be, say, employee scheduling. The part needing customisation might be the part that determines whether a given employee can fulfil a given time slot. It's customisable because we might have a variety of different kinds of employees, time slots, and constraints that determine whether or not a given employee can work a given time slot.

// In this example, customisableBit is a function.
//
// Alternatively, it could have been a string representing
// some snippet of code to be EVAL'd, but then it would have
// to be syntax-checked and/or compiled n1 * n2 times.
// Do we really want to be performing O(n^2) worth of syntax checks
// and/or compilations if we don't have to?
//
function complexCustomisableAlgorithm(customisableBit) {
   ...complex but non-changing stuff...
   for (i=0; i<n1; i++) {
      for (j=0; j<n2; j++) {
         customisableBit(i, j, somedata)
      },
   },
   ...complex but non-changing stuff...
},

// Here's a single implementation of the customisable bit
function customisedBit(x, y, data) {
   ...customised bit...
},

// Invoke a single customisation like this:
complexCustomisableAlgorithm(customisedBit)

Now what if we need to customise based on some dynamic data, i.e., we require a data-dependent series of customisations? (In a business context, this might mean (for example) generating a set of schedules, each one varying in some tuning parameter.) So, we need a HOF that returns a function. Like this:

// Return a customised (i.e., dependent on p and q) function
function customisedBitMaker(p, q) {
   return function(x, y, data) {
      ...customised bit that needs x, y, data, p and q...
   },
},

// Create and invoke them like this:
for (int p=0; p<q; p++) {
   customisedFunction = customisedBitMaker(p, q)
   complexCustomisableAlgorithm(customisedFunction)
},

Re:"Do we really want to be performing O(n^2) worth of syntax checks and/or compilations if we don't have to?"

Machine labor is usually far cheaper than human labor. And an EVAL version may be easier to "X-ray" to debug and log.

You appear to be stating that you believe the generation of a function inside a function to require more expensive developers than not generating a function inside a function, and that debuggers break when they encounter higher-order functions. Is that what you believe?

table: scheduler
-----------------
title
processString   // command or Eval string
processType     // indicates whether a command-line or functional call via Eval
timerType       // hourly, weekly, Nth x-day of the month, etc.
timerParameter1  // time, day, etc.
timerParameter2  // 2nd parameter, depends on the timer type
logResults      // TRUE if logging process output
problemNofify     // eMail addresses to notify if problems (optional)
finishNotify      // eMail addresses to notify when finished (optional)

Window 1A:
sales_stats.php?time_range=month&sort=location&report_type=compact&etc=1234

Window 1B:
sales_stats.php?time_range=day&sort=location&report_type=compact&etc=1234

Window 1C:
sales_stats.php?time_range=day&sort=location&report_type=detail&etc=1234

Window 2A:
sales_stats.php?time_range=month&sort=product&report_type=compact&etc=1234

Etc...

Are you looping through each possibility? What if the problem is too big such that we want to use a genetic algorithm?

No problem. Exactly the same approach applies. I have developed GA-based schedulers using precisely this idiom.

And the fitness calculation may require indexes/lookups to be sufficiency efficient. Pre-processing certain steps in aggregate may also make certain calculations more efficient. The assumption of an isolated calculation function may keep us from using Economies of Scale such as pre-sorting or pre-aggregating. It may be one of those "assume a spherical cow" moment. SeparationOfConcerns is often a myth or excessively ideal in the biz world.

It's perfectly reasonable for the scheduling algorithm implementation to pass (possibly mutable) data about the state of the scheduling process to the fitness function via a parameter. The 'somedata' argument in the code above is intended to illustrate that.

Achieving better SeparationOfConcerns in the face of problem complexity are what facilities like HOFs help us achieve. Indeed, much of programming -- and advances in programming techniques -- are about achieving looser coupling, tighter cohesion, and better SeparationOfConcerns.

I'm not sure why Eval expressions are out of the question performance-wise. A page full of independent sub-panels (Brady-Bunch-opening-style) isn't going to tax Eval processing. As far as I can tell, JavaScript's built-in DOM timer requires a HOF, and so we are stuck with that no matter what. But that doesn't mean we cannot manage specific processes using some other way, such as Eval expressions.

It may not be efficient to have a timer process for every panel under ANY approach, but rather go with a timer process (thread?) for each interval. For example, suppose we create one timer process each for the refresh intervals of 0.5, 1, 2, 4, 8, and 16 seconds. Thus, any new panel will be assigned to one of these six intervals, and any given interval may have multiple processes assigned to it. (There are different ways to allocate panels, but I'm fairly arbitrarily choosing a simple one for illustration purposes. The names chosen for panels here fit spreadsheet cell referencing conventions.)

This can be implemented by having 6 arrays for each interval. Adding a new panel could resemble:

// Add a new panel (pseudo-code)
// Parameters: panel ID, expression (for Eval), and the timer interval name
documentX.addPanel("B8", "foo(123);", "eight");

This would assign panel B8 the process of "foo(123)" to re-execute every 8 seconds so that the process for panel B8 gets executed every 8 seconds.

"addPanel()" would put the string and panel ID into a table-like array called "intervalEight":

intervalEight[rowID]['target'] = panelID;   // B8 in this case
intervalEight[rowID]['expressn'] = evalExpressn;  // "foo(123)" in this case

Every eight seconds, the eight-second timer process loops thru the "intervalEight" array and does an Eval on each expression and replaces the panel (a DIV or a frame) with the resulting HTML contents.

If 5 panels use eight-second intervals, then five Eval's are executed during each cycle. I don't see why this would be a resource problem. For our shortest interval, the half-second cycle, if we had 5 panels under this, then that's still 10 eval's every second, which is nothing to write home about still. If we have 5 panels each in all six interval categories, then that comes to about 25 eval's a second (and a rather crowded screen with 30 cells). If we had say 500 Eval's per second, then we are risking resource problems, but I don't see anything approaching even 100 per second for this kind of app unless something really unusual is being done.

It's probably not as efficient as using HOF's to "register" each process, but I don't see it being a bottleneck for a panel refresher browser-based app. If it needed video-game-like refresh rates, then we may have issues. But in that case, an org would probably want desktop applications and hire a video-game developer or the like, not a generic custom biz app developer.

This is not necessarily saying that using Eval is superior, but it appears to be a competitive alternative for the multi-updating-panel situation.

Assuming we don't consider the battery-power and CPU constraints of mobile devices -- for which any avoidance of processor-heavy compilation and/or syntax checking (which eval must do) is considered beneficial -- what do you gain by doing the above as opposed to the HOF-based approach?

Also, what do you gain by assigning '"foo(123);"' to some variable 'expressn' -- presumably to be executed later via eval(expressn) -- as opposed to assigning it 'function() {foo(123)},' and executing it as expressn()?

Not much here. I never claimed Eval (or other competitor techniques) were always a significant improvement over HOF's. Please review the initial claim in the chain. But I would point that mobile devices usually don't have room for 50 cells on their screens unless you are using micro-fonts. (I doubt they have the bandwidth to pull info from that many sources quickly anyhow.)

Why are you assuming the display is textual? It can easily consist of coloured graphs and the like. Mobile monitoring dashboards are increasingly popular, and whilst every effort is made to minimise bandwidth consumption by reducing over-the-wire data transmission, the value of having live monitoring is generally considered to be worth the cost of 3G/4G/WiFi.

Evals do have the advantage of being easy to display as text during debugging, and storing in cross-language tools, such as a database. The requirement didn't come up here, but it's one of those things that depends on the org's or app's particular details.

HOFs have the advantage of being source code, and thus are easy to display as text even during development. During debugging, they are no more difficult to debug than any other function. As for storing in cross-language tools, such as a database, that's almost never a requirement, though I've shown it can be done at http://dbappbuilder.sourceforge.net/docs/AnonymousAndFirstClassOperatorsInTutorialD.pdf Most end-user requirements demand data in the database; code is for the developers to worry about.

I already agreed during the "weather example" that there are occasionally times where hardware may make the difference in technique choice (although weather wasn't a biz app). In this case, you seem to be pushing for an extreme example in order "force" a hardware-centric requirement. Yes, if A happens and B happens and C happens at the same time, then it may make HOF's the clear winner. But nobody is looking for blue-moon exceptions to the rule because we all agree they exist. And HOF's have their own blue-moon problems. Thus, we have cases where HOF's run into problems, cases where both HOF's and Evals work just fine, and cases where Eval's won't work. (There are also non-Eval competitors to HOF's, as a reminder.) There frequency where only HOF's are a decent choice in biz apps appears to be rare, and your chosen scenarios don't seem to be making a good case that they are not rare. -t

HofPattern actually shows up frequently. Its apparent rarity is only because among the popular languages typically used in business application development, only JavaScript explicitly supports HOFs, so most programmers inevitably think in terms of alternatives rather than even considering HOFs. Of course, ObjectOriented languages often support creating FunctorObjects -- either explicitly or implicitly -- or trivially allow passing an object (and, therefore its associated methods) to some algorithm, and these are obviously used extensively in modern application development. It's only a matter of time before more languages will explicitly support HigherOrderFunctions -- this is inevitable -- which will result in the majority of developers recognising their efficacy.

You keep claiming "it" shows up frequently, but cannot seem to find very good examples, instead pushing into environment-centric restrictions. A decent GUI tool would give us better timer choices. This example is a slave to limitations/issues of the client-side environment, not about organizing custom business application code.

I've given examples: Timetabling, employee scheduling, logging. It also applies to costing, forecasting, logistics route-planning, or anywhere there is an indivisible algorithm that needs customisation -- i.e., HofPattern. However, popular business application development languages don't support HOFs, so we're not used to using them. We're used to either using workarounds, or using object-oriented approximations.

In VB-classic, I used to be able to add timer code by dragging a timer widget into the form, right-clicking to set the cycle duration (and other optional factors), and then double-clicking to open up the code snippet edit box and type something like:

myProcess01(7)
if foo.setting < 4 then
   myProcess02()
   statusBox.value = "done with 2"
end if
logThingy(99)

And then hit the save/close button. The timer then executed that snippet every cycle during run-time. (Multiple timers were possible per form.) Whether it used HOF's, objects, or gerbils under the hood was usually of no concern to me, the custom app developer. VB-classic sucked in many ways, but let's give them Kudos for what they did right (or for stealing the best ideas).

In textual form, it would ideally look something like this:

<!-- example: pond-72 -->
<timer name="timer01" interval="2.5" unit="seconds">
  myProcess01(7)
  if foo.setting < 4 then
     myProcess02()
     statusBox.value = "done with 2"
  end if
  logThingy(99)
</timer>

How is that different from, say, this?

timer01 = startTimer(seconds(2.5), function() {
  myProcess01(7)
  if foo.setting < 4 then
     myProcess02()
     statusBox.value = "done with 2"
  end if
  logThingy(99)
},)

class myTimer inherits timer
  constructor
    interval = seconds(2.5)
  end constructor
  function main
    myProcess01(7)
    if foo.setting < 4 then
       myProcess02()
       statusBox.value = "done with 2"
    end if
    logThingy(99)
  end function
end class
// The more compact initializer version is not shown

I'm not sure what the VbClassic code-generation wizards and form painters have to do with this, but it's notable that event handlers are ideally implemented using HOFs.

Even if that was true, why should I, as a custom app developer, care? That kind of detail is a tool implementation detail that should be handled under the hood from my perspective. We are going backward in technology with primitive HtmlStack crap, which is the only reason it's a half issue.

Why should you care? Because you are, I presume, a programmer. These are the things programmers care about. If you don't care about programming, but (at best) endure it and ideally want all programming to be eliminated, then these discussions probably aren't for you -- you're looking for tools that not only eliminate HOFs, but which eliminate functions in general, loops, and IF statements too. Otherwise, why hide HOFs but not, say, loops?

There are MentalMasturbation projects we do on our own time, and there is being productive at work by not having to concern one self with minutia not related to the task. I understand that a given tool may prefer or even force it to be done one way or another, but that doesn't make it some kind of universal truth of superiority. If a given UI tool prefers HOF's, I'll consider HOF's. If it prefers OOP, I'll consider using OOP. If it prefers gerbils, I'll consider using gerbils. But that's missing the main target here: about how to better write and manage biz code, NOT how best to work with JavaScript or BrainFsck or whatnot. -t

Fair enough. I think what's been made abundantly clear is that HOFs are of value, even in custom business applications. When more business application development languages explicitly support HOFs, then we'll see that even more clearly.

"Support" and "force one to use" are not the same thing. You still haven't made a real case for them in terms of general code organization, rather relying on some specific API to "justify" them.

Eh? That very justification, "in terms of general code organization", is precisely what this page is for. It presents a general case for using HOFs to inject customisations into otherwise-indivisible implementations of algorithms.

You didn't compare it to alternatives, including alternative API's that one may be stuck with using due to circumstances, and didn't count or measure anything objective. If this is the best case you can make for HOF's for custom biz apps, then actually you've only made my case because you claim to have extensive C.B.A. experience and a whole world of scenarios to choose from, and this rigged thing is the best you can do. Thank You. --top

How is the above "rigged"? I have pointed out that the alternative is to inject an object, but that's essentially conceptually identical to a HOF. It adds only syntactic overhead to achieve the same end, which is to carry an external function (or possibly functions, in the case of an object) that carries some external state into the function that implements the algorithm. There are no other alternatives without the pattern itself disappearing. For example, you can partition the algorithm (which means it's no longer considered indivisible!) but for indivisible algorithms -- exemplified here using a double nested loop -- this obviously results in negative consequences. This is all obvious; I'm not sure what I can do to make it clearer. There's no need to count or measure anything, because the implications are self-evident. Anyway, what would I count? Inject a function = 0 awkwardness, but being unable to inject a function (or object) means partitioning the algorithm = 1 awkwardness? I'm certainly not going to iterate through every possible individual case -- that would be ridiculous -- hence my provision of a general pattern.

I'm not sure what you are talking about. Your head seems to be caught in the machine end of things. We are not talking about how to make interpreters/compilers. Why should it matter to me that it's "conceptually identical to a HOF"? The fact that you cannot measure anything of utility to a CBA developer is telling.

I'm not talking about "how to make interpreters/compilers", but how to implement algorithms -- you know, those things that custom business applications are made of, that do things like calculate timetables and employee schedules and figure out costing and determine vehicle routes and stuff -- so that the algorithm can be an indivisible unit for maximum performance and simplicity, but you can still customise it at run-time by injecting the customisation as a HOF (or object). I'll give you a basis for measurement: How would you handle that situation without HOFs or objects?

What situation? Are we starting a new scenario?

No, the same situation as HofPattern, i.e., an indivisible algorithm requiring customisation. How do you handle the customisation without HOFs or objects?

It depends on the details of the business requirements and environment.

No, it doesn't. This is an abstraction. And re-engineering the application so that you don't need the algorithm any more, or that you can buy the solution pre-made, is not a reasonable answer.

Not a good one. If you can't quantify the alleged improvement, then you are going to have a hard time being convincing. You may not like science, but it's good for you, like broccoli.

Many engineering considerations are not easily quantifiable but are trivially qualifiable. For example, we can easily see that structured code is qualitatively superior to using GOTOs, but this is not easily measurable. Regarding the pattern shown on this page, fortunately the case is easier to make -- there simply isn't a reasonable alternative. Historically, a typical procedural (i.e., non-HOF, non-object) solution would have required inefficiently (and probably awkwardly) partitioning the algorithm's implementation, or implementing it in such a manner that customisations would have to be embedded in it. Either is qualitatively inferior to injecting HOFs or objects into the algorithm's implementation. Obviously, experiments can be constructed to quantitatively verify this, but one can do a trivial gedankenexperiment to show it too. For example: How many times would the algorithm's implementation need to be changed to suit new scenarios? With HOFs and objects, it wouldn't -- customisations can be injected. Without HOFs and objects, it would have to be changed once per new scenario. If the number of new scenarios is 'n', then with HOFs and objects the number of algorithm implementations == 1, without HOFs and objects the number of algorithm implementations == n. 1 <= n. QED.

You haven't shown realistic change scenarios. You go out of your way to lodge the scenario into unlikely, vendor/client-specific corners, or assume one aspect is very stable while another is very dynamic without describing why it's that way in the environment.

Really? Can you show how and where I have made these mistakes?

Are you familiar with the C/C++ standard implementation of qsort? It's the quintessential example of what I'm describing. The generic, indivisible QuickSort sorting algorithm is implemented as a function qsort(), but it requires customisation: the comparison operation varies depending on the data type being sorted. So, the comparison operation is injected as a higher-order function. This means the qsort() implementation never needs to change; only the comparison operation changes to suit any scenario for which QuickSort is appropriate. HofPattern is a generic abstraction of this concept. Why do you think qsort() is implemented this way? How would you implement it so it doesn't use a HOF, and what are the implications?

And there are plausible WetWare theories on goto's versus blocks even if they are not quantifiable (I've written my own observations on goto's elsewhere). But I would never insist that such theories are iron-clad and insult those who claim they function well with goto's. What makes my WetWare happy may not apply to others. If you ain't got the science, man up and admit it.

It is almost impossible to sustain any reasonable engineering argument in favour of using GOTOs over structured programming, at least that doesn't devolve into supporting a style of programming favoured by at best a rapidly-vanishing handful of "old skool" developers.

And I have already agreed that HOF's may be a more efficient option in some situations, machine-performance-wise, but that it's usually not a bottleneck in practice for custom biz apps. I'm happy to make the machine slave away so that the human doesn't have to.

As illustrated with HofPattern, not only are HOFs more efficient machine-performance-wise, in some cases there is no reasonable alternative.

If the API forces you to use it, then yes. If an interface requires SQL, then one must interface with it in SQL. If it required Mayan hieroglyphics, then one must use Mayan hieroglyphics to interface. This should go without stating. This is getting frustrating; I'm holding back angry statements that are not meant for family viewing, but the pot is near boiling and sputtering out the edges. In my humble opinion, your scenario is rigged and we keep making the same arguments over and over. Unless you find something quantifiable and not tied to exclusive products, there will likely be no closure here.

Why are you reacting emotionally? Is it because your anti-HOF stance is emotional rather than practical?

What do you mean by "tied to exclusive products"?

How is my scenario rigged? And, if you feel it is rigged, do you feel qsort() in the C/C++ standard library is rigged as well?

This is more than just an API that forces you to use it. QuickSort (as implemented in the C/C++ standard library) requires customisation to be useful -- it needs a "compare" operation specific to the data being sorted -- and it is difficult to partition. If it could be partitioned easily, you could divide it into separate, semi-independent "blocks" and interpose customisations between them. Unfortunately, QuickSort -- like many algorithms -- defies being broken down into multiple semi-independent blocks. So, what are the alternatives?

Given the above, is there any reason not to choose the last option?

It's never been an issue in all the biz apps I've worked with. If the existing collations are not good enough, I create an additional column(s), real or virtual, to fine-tune the sorting using compound-column database sorting.

Indeed. Unless you write business applications exclusively in C or C++, you're unlikely to have used QuickSort. QuickSort is a specific found-in-the-wild example of the general HofPattern. Precisely the same principles and conditions that apply to QuickSort apply to a wide variety of business-oriented algorithms. Instead of QuickSort, it could just as easily be employee scheduling, timetabling, logistics routing, payroll tax calculations, costing, sales forecasting, or some other business-oriented algorithm. Of course, most of the popular programming languages for business application development -- except JavaScript, if you're doing client-side Web development -- simply don't support higher-order functions. That means of the four alternatives above, the last one is usually unavailable. So it's not surprising that you prefer the first three and are sceptical of the last one; it's almost certainly unfamiliar and therefore outside of your development comfort zone. Of course, that will inevitably change, as the value of higher-order functions -- for precisely the sort of pattern described on this page, among other things -- is virtually undisputed in the programming community, regardless of domain. So, we will see HOFs in future business application development languages.

You keep claiming that, but fail to show semi-realistic scenarios (a business setting) of them helping a lot for custom biz apps. The above looks pretty much to me like a repeat of claims already made. Ideally at least 3 biz sub-domains should be demonstrated, but I'll settle for one at this point as a start (not tied to specific API's or hardware). We just seem to have very different ideas of what we consider and/or accept as "good evidence". I don't know what to say. We are at an "evidence impasse" that seems unbreakable and are going around in circles repeating the same arguments hoping they finally stick a 7th time around or whatnot. I won't stop believing that GoodMetricsProduceNumbers (if the claim is intended to be "objective" [1]), and you won't stop believing in (what looks to me like) ArgumentByElegance. This issue is at a higher level than that of HOF's themselves. I suggest you encourage a different WikiZen with a different evidence presentation style to demonstrate the power of HOF's in CBA's. -t

I wish it were as simple as ArgumentByElegance! It's quite simply the case that for some indivisible algorithms, the four alternatives above are the only alternatives. Employee scheduling is a good example, as it typically consists of nested loops and needs to invoke a customisation -- the fitness function -- in the middle of them. So, much like the qsort() example above, in the simplest case the scheduling function signature winds up being something like schedule(Employees e, Slots s, FitnessFunction f) where 'f' is a function that returns a weighted indication (and sometimes just a boolean) that tells whether or not a given employee can fulfill a given slot, and f can vary under external conditions like whether we're generating a regular schedule or an overtime schedule. And, of course, the schedule() algorithm implementation, once written, gets used on a number of projects. In other words, it's a characteristic example of HofPattern. My other examples are exactly the same: an indivisible algorithm, and a need for customisation.

I'm skeptical because biz logic often involves lots of business "sub-parts". They cannot be easily summed up into compact "functions"; and managing the repeating patterns (commonalities) are often best served with something closer to set management (SetTheory) and/or nested IF statements: interweaving features or characteristics selected in a buffet manner, which often leads to declarative "switches", not functions. A list of functions or function-only-based interfaces is usually too blunt an instrument in such a setting. The granularity of functions is too large. In practice the useful variations or their representation will be on a sub-function level, perhaps closer to the parameter level. Something like PredicateDispatching is a better fit, but these usually leads to a single "god function", in which case HOF is no longer helpful since there is only one. Further, power users are often going to manage most of the variations/combo's via something akin to a RuleBuilderInterface for non-trivial implementations, not programmers writing functions. Conceptually it's a great idea that I wished worked, but just doesn't float in typical CBA settings. Maybe you need to invent "HOP" - Higher Order Parameters :-) You seem to be making some of the same mistakes that "taxonomy fans" sometimes make, thinking that the variations can be nicely mapped to a hierarchical taxonomy (sub-types) because trees are such a "clean and pure" organizational concept. Functions are potentially even "bluntier" than trees because trees at least have sub-trees in them for finer tuning. (Regarding employee time-slot scheduling; again, I don't have enough knowledge of that particular sub-field to comment or demonstrate. I suspect most co's buy pre-made software such that it's no longer in the CBA realm.)

How do you know you that your perceptions -- i.e., that repeated patterns are best handled with "nested IF statements", or that features should be selected in a "buffet manner", or that functions are usually "too blunt an instrument", and so on -- aren't simply the unfortunate result of your deliberate avoidance of certain programming techniques -- like higher-order functions, inheritance and polymorphism, and so forth -- that make it possible to efficiently, effectively, elegantly, and re-use-ably handle the complexity of programming custom business applications, regardless whether you're developing them in house or to sell to other companies as pre-made software?

Couldn't the same principle potentially apply to you with regard to dynamic languages (such as Eval) and database usage? If I am self-deceiving myself, I'm not aware of it. And you could fall victim to the same human weakness just as well. The best way to settle it is to create semi-realistic coded examples or scenarios, and then let the reader decide how relevant those examples and comments on those are to their own world. HowToSellGoldenHammers still applies. And I do agree that some OOP concepts have their place; it's just not as wide as many OOP proponents claim or used to claim (OopNotForDomainModeling). Yes, it's "nice" to have HOF's in one's programming toolkit, but I see no significant improvement from them for CBA, only minor incremental improvements for a narrow set of apps.

Could the same principle potentially apply to me? Could I fall victim to the same human weaknesses that all humans do? Could I be blinded by my own biases or preconceived notions, or be limited by comfortable familiarity, or let laziness cause me to roll back from steep learning curves and lose out on something helpful? Absolutely! It's why I continually read about new programming techniques, new development processes, and new programming languages, and try them whenever possible. It's why I continually revisit existing techniques, processes and paradigms to see if there's something I might have overlooked. It's why I search for and read published summaries of research into all aspects of SoftwareEngineering. It's why I constantly reflect on and re-evaluate my own programming, and ask myself whether my code is as readable, as maintainable, and as efficient as it could be.

As a result of that process, I've come to realise that HOFs are superior to evals, embedded case statements, or the first three alternatives I listed above. There are no downsides. The only objection I can possibly see is that HOFs are, for many programmers, something new that needs to be learned. But once learned, they're as easy to understand as any function -- or, for that matter, any other programmatic construct, like conditional statements and loops. Of course, every neophyte programmer had to learn about loops, and many find them to be the first big conceptual learning hurdle in programming. Is that a reason not to use loops? Similarly, because HOFs may involve a small learning hurdle, is that a reason not to use HOFs?

I would note that when I do start out using something very similar to HOF's, such as Eval'd expressions or SQL clauses, if there becomes many instances (variations on a theme), then certain patterns start to appear such that a declarative interface(s) starts to become apparent such that table switches/flags or configuration screens or RuleBuilderInterface techniques start to become of more utility, and gradually more of the management and creations of variations can be handed off to power users or junior programmers. If there remains only a few variations, then the technique used doesn't matter much because usually a low quantity of variations also means a low quantity of changes in the variations such that all the usual techniques either score similar, or are not a significant point of change anyhow because they contribute relatively little to the change pattern "scores" of the entire application. In short, with few variations, most of the technique choices are a wash or insignificant to the big picture. If there are many, then attribute-driven or declarative interfaces are best applied so that power users instead of programmers manage most of them. Now I may not know much about employee shift scheduling software techniques (mentioned above), but I'd be willing to bet that if we study the pattern of actual scheduling techniques variations in the field, the above dichotomy pattern would show itself. Eval'd expressions and SQL clause lists/tables are often quite useful for the prototyping stage, but as the app matures, less so.

Typically such medium-sized "expression lists" eventually evolve into roughly 3 to 12 "strategies", where each strategy has its own set of parameters/fields/switches that often differ in quantity and nature from other strategies. Sometimes some combo's of strategies are not mutually exclusive, which may complicate the UI a bit. In bigger variation pools/apps, there are often interweaving overlaps such that there is more of a feel of mix-and-match than the strategy->fields hierarchy of its smaller cousin. At this point, some kind of RuleBuilderInterface may be more appropriate if you don't want a combinatorial explosion of sub-screens.

Sorry, I don't see how the above represents an alternative to HOFs. You appear to be referring to broad architectural issues, which HOFs are not -- at least, not necessarily. They're often a small-scale technique on the same level of detail as conditional statements and loops. They're merely a way of injecting customisation into an otherwise closed place. E.g., you've got something like this:

function somethingUsedFrequently() {
  for (i=0; i<n; i++)
     for (p=0; p<i; p++) {
         // ...bunch of code goes here...
         // what happens here depends entirely on where somethingUsedFrequently is called
         // ...another bunch of code here...
     },
  },
},

Do you really want to be change somethingUsedFrequently() every time it's used in a new place? What if it's used slightly differently dozens or even hundreds of times? What if the customisation is parametric, i.e., generated from some dynamic value? Is it better to do this...

function somethingUsedFrequently(algorithID) {
  for (i=0; i<n; i++)
     for (p=0; p<i; p++) {
         // ...bunch of code goes here...
         select on algorithID {
           case "foo": fooFunction(p)
           case "bar": barFunction(p)
           case "glif": fooFunction(p) + barFunction(p)
           case "bwep": barFunction(p) + 73
           otherwise: myDefault(p)
         },
         // ...another bunch of code here...
     },
  },
},

...or is it better to do this?

function somethingUsedFrequently(algorithFn) {
  for (i=0; i<n; i++)
     for (p=0; p<i; p++) {
         // ...bunch of code goes here...
         algorithFn(p)
         // ...another bunch of code here...
     },
  },
},

Note that in the second example, somethingUsedFrequently() need never change. It can be embedded in a library and remain constant, regardless of context. In the first example, somethingUsedFrequently() will need to be changed every time it's used in a new context. Furthermore, the functions passed to somethingUsedFrequently can be generated by a function, like this:

function algorithGenerator(x, y) {
   return function(p) {
       return splorb(x) * spleen(y) / p
   },
},

for (q=10; q<10000; q++) {
   fn = algorithGenerator(q, rand())
   somethingUsedFrequently(fn)
},

Try that with case statements! I can't see any reason why case statements or eval'd expressions would even be considered as an alternative. The last example is so obviously superior -- and so obviously difficult with anything other than HOFs -- that I find it difficult to imagine how any counter-argument can be sustained.

Finally, to complete the equivalent to your case statement example, you'd probably have this:

function glif(p) {
   return fooFunction(p) + barFunction(p)
},

function bwep(p) {
   return barFunction(p) + 73
},

The above would be used like this:

somethingUsedFrequently(fooFunction)
somethingUsedFrequently(barFunction)
somethingUsedFrequently(glif)
somethingUsedFrequently(bwep)

Note that no case 'default' is needed, nor are run-time string comparisons needed to determine which operation to perform. Is there any benefit to the case-switch or nested if/elses here?

As a refresher, co-competitor techniques to HOF's include Eval'd expressions, conditionals (case-switch or nested IF/ELSE's), SQL clauses, OOP sub-classing, and RuleBuilderInterface(s). They are not necessarily exchange-able one-to-one, for in some cases multiple techniques may carry the load that is done all in one technique in another case. -t

Only eval'd expressions are a competitor. Injecting objects (is that what you mean by "OOP sub-classing"?) is equivalent to HOFs; it's not a competitor, it's a variant. RuleBuilderInterface(s) will need to be evaluated in code; that code is likely going to be in the function passed to some algorithm. Conditionals are obviously inferior in every respect, and I don't know how SQL clauses would apply; are you familiar with the overhead that invoking SQL queries represents? And, eval'd expressions are obviously inferior, for reasons noted above.

Saying sub-classing or "injecting objects" is "equivalent to HOF's" is something I disagree with. How do you define "injecting"? Anyhow, OOP is supported by most mainstream languages now such that your claim that people don't use HOF's because mainstream languages often don't support them appears to be a contradiction. And are you claiming that a typical RuleBuilderInterface is usually best done with HOF's?

Sub-classing and injecting are orthogonal. Sub-classing is about creating class definitions. "Injecting" is shorthand for "passing in from outside for use inside." HOFs and injecting objects are equivalent, but not the same. A function passed as a parameter typically implicitly captures its execution context as a closure; passing an object typically requires explicitly constructing that object with the desired context. In short, passing objects and HOFs can achieve the same goals. However, objects typically require more programming effort and HOFs are syntactically (and conceptually, for the most part) distinct from objects, so object-oriented programmers don't typically think in terms of HOFs even if they're explicitly creating FunctorObject's.

I'm not claiming that a typical RuleBuilderInterface is "usually best done with HOFs". I do claim that it's entirely appropriate and reasonable for a RuleBuilderInterface to be evaluated at run-time by some function, and for that function to be passed (injected) into another function, but that doesn't necessarily mean it should always be done that way.

Also, you don't seem to dispute that Eval can be used to create HOF-like behavior. I pointed out that I've used such on occasion, but mostly just for prototyping because when requirements settle down they are often best replaced with something else, and it's not for reasons of "type safety" and compiler checking (I prefer dynamic languages anyhow). Your claim that I don't try HOF's is thus diminished.

If you only use eval to create HOF-like behaviour, then you've tried HOFs precisely to the extent that someone who only eats worms can claim to have tried spaghetti.

For your employee shift scheduler, are there really so many algorithms that we need HOF's to manage them? What's wrong with a case statement:

fitnessDispatcher(algorithmID, p) {
  select on algorithID {
    case "foo": fooFunction(p)
    case "bar": barFunction(p)
    case "glif": fooFunction(p) + barFunction(p)
    case "bwep": barFunction(p) + 73
    otherwise: myDefault(p)
  },
},
// I'm not using C's "switch" syntax because it's an abomination

We just have to add another CASE for a new algorithms/function.

Curious. It is precisely that "just have to add another CASE" that I try to avoid. Every change to the 'fitnessDispatcher' function, no matter how innocuous, risks breaking it or inadvertently altering one of the other cases. Why do that, when using HOFs eliminates that possibility entirely? What is gained by case statements? What do you lose with HOFs?

Generally I find that such lists almost never get long unless the API or user interface is poorly factored. If you factor out the commonalities, then it grows into nested IF's and/or non-mutually-explicit behavioral conditions based on parameters or info from the database, etc. I don't know why this pattern happens, but it just does. Complexity just simply is rarely constrained to one dimension. It, eventually squeezes out into other directions or dimensions (such as parameter differences) if it keeps increasing. Rarely does one aspect get all the complexity or quantity increase while other aspects stay relatively sparse or simple.

You can prove me wrong by showing actual long lists and what each algorithm is. You don't necessarily have to implement each one, just describe what they are, such as "Algorithm X from Dr. Snaffew from Flubber U, Journal of Zog, May 2004."

I agree that with say something like sorting algorithms, there could be 50 or more algorithms in the wild. However, for a custom biz app, we may select a handful to cover a sufficient variety of performance and/or results trade-offs. The users will get confused if we list 50 in the pull-down list. Reality often culls extreme cases in such a way. GoldPlating is possible but often not wise nor budgeted. (In practice with CBA's, I usually use the database to sort and rarely have to do it in app code, but I'll humor the example for now.)

It's not the algorithms that vary significantly, it's the customisations that need to be injected into the implementations of algorithms that vary. For example, Dijkstra's Algorithm is the same whether it's finding the shortest path -- for, say, logistics planning purposes -- over a set of roads or a set of canals. However, we may wish to inject a customisation into the implementation of Dijkstra's Algorithm to allow for the fact that it's being used for roads in one context, and canals in another. Or whatever. The important thing here is the concept, not the specific case.

Another example you often use is the comparing function for sorting algorithms. In practice it doesn't make sense to have gazillion functions, but rather have a collating sequence defined in a database or map file (character or symbol, and its sequence rank). You then add another parameter--the collating sequence name or table name--such that a lot of complexity becomes "absorbed" by (shifted to) the new parameter so that we don't need gazillion new functions and are thus not changing the CASE list very often (if that's what you fear). In theory you could use gazillion functions, but it's a case of AttributesInNameSmell. Most CBA scenarios, if factored properly, will grow some "width" with any increase in complexity, and the length (quantity of functions) growth will slow down. -t

The comparing function varies depending on the types being compared; i.e., one comparison for unicode ISO-8859-5, another for ASCII strings, another for integers, yet another for floating point values, yet another for scholastic exam grades, another for the lightness or darkness of baked bread, another for colours, etc. In other words, the collating sequence varies by type.

Okay, but a CUSTOM biz app is not going to need to be everything to everybody; it's usually going to be called on to do a specific job. (Again, databases are usually used for most sorting and have done the job just fine. This would be for some rare, special, and so far unidentified project.)

If it's just sorting, perhaps. Remember, the "comparing function for sorting algorithms" is an illustration of the general HofPattern.

Yes yes. We are in dire need of a semi-realistic specimen/scenario so we can move beyond toy or lab examples.

How would you refactor this into HOF's?:

// Example "Pete-82"
function foo(title, flag1=true, flag2=false, flag3=true, flag4=true) {
  // values after are defaults if explicit is not supplied
  behaviorCommonToAll(title);
  if (flag1) {
     if (flag3) {
       blah(title, flag4);
     },
     moof();
  }, else {
     if (flag2 && flag3) {
       znig();
     }, else if (flag4) {
        grob();
     },
  },
  if (flag4) {
     fible(title, flag1);
  },
},

I wouldn't. I'd get rid of foo() -- as it appears to be trivially divisible, and therefore not an example of HofPattern -- and compose functionality as and where needed from behaviourCommonToAll(), blah(), moof(), znig(), grob(), and fibble(), because it appears flag1, flag2, flag3, and flag4 are essentially synonyms for which functions you want to call. Why not simply call the functions you want to call?

They are not synonymous because they affect different parts in different ways. And typically the feature flag and controller values come from the database, where user and power-user preferences and content attributes are stored. It's not in code such that it has to happen at run-time.

That's dynamic dispatch based on decision tables and the grammar represented by the user interface. As noted, there isn't an indivisible algorithm here, so HofPattern does not apply. [[I added emphasis, -t]]

Well, that's the most common general pattern of CBA's I see, which reinforces my suggestion that HOF's are not very helpful in CBA's.

That suggests the "CBAs" you write are primarily data-in/data-out and summarisation, sometimes called "reporting apps". Nothing wrong with that; there's a large market for such things. It also means you are, indeed, unlikely to need HOFs. You can probably build everything you need out of SQL and a procedural scripting language with conditionals, loops and procedures.

They are a variety of things, not merely "reporting apps" or I/O. It's a lot about "routing" in that the attributes/settings (as stored in the DB) control when and where things "show up" or are processed. For example, an input screen may have a flag for "This foo falls under regulation/policy X". That switch's value may affect a half-dozen or so conditionals along the business process chain, controlling whether certain actions are taken or not taken including which reports it shows up on, how it's categorized (including category validation/review), how it's billed & taxed, how long it's kept in the archive, etc.

But at least we seem to be coming to some semi-agreement, which is rare around here. --top

It sounds like there isn't a lot of (or any?) algorithm implementation, at least not in the SoftwareEngineering or ComputerScience sense. Depending on what development tools you use, you might see HOFs being used (either now, or eventually) in event handlers to implement callbacks, but outside of that, there's not much justification for them.

It depends on how you define "algorithm". All that damned code is doing something. Perhaps you lean toward certain niches or specialties that shape your view of what CBA "looks like", and vice verse. We all only live one life and cannot easily hop into others' shoes. CBA's do tend to be a little bit of everything: a little bit of parsing, a little bit of semi-AI, a little bit of statistics, etc. However, most of the time, none of these algorithm-centric processing portions are big or involved enough to use the kinds of patterns you seem to be talking about. You don't implement 15 different optimization algorithms; for management picks their favorite and you implement one or two and it stays that way for a good line time. If it does become a "big deal", such as a line-of-business process, then specialists are then brought in or commercial software is purchased for that sub-niche. -t

Yes, I'm well aware of the "specialists are brought in" factor, having been one of the specialists brought in on various projects of this sort. The algorithm-centric processing portions I've worked on have used the patterns I'm talking about precisely to avoid having to implement any given algorithm more than once, and that's independent of how "big or involved" they are.

Perhaps. I didn't see the specs, change analysis, etc. for those, so I cannot comment. Most of the times such comes up, I've seen it done by purchasing/renting existing software, not custom solutions. Maybe your niche is those semi-rare cases where no commercial product is available that fits sufficiently. I'm a "rank and file" CBA developer for the most part and see "typical" requests from that perspective. I've worked with specialists such as statisticians, accountants, scientists, engineers, GIS (mapping), and even an AI expert once. These are usually special needs that generally require more specialized experience and/or degrees. If the org asks me for something outside of my expertise, I'll gladly suggest they find a specialist. I don't say, "I wont do it", rather I tell them it's new territory for me such that there is risk of failure. Sometimes they let me experiment, knowing there's no guarantee, sometimes not. I just state the risks as best I can. -t

That's just based on my experience and hanging around other CBA's, who fulfill a role similar to mine, but in different organizations or departments. The specialists tend to be either in-house specialists, such as statisticians and GIS experts; or they are "rented" from the outside for say special high-speed communications systems.

More on the Pete-82 example (above) in SwitchCaseListVersusHof.

And if the GUI/UI/Webpage kit is done right, then the fact that the event "snippet" (EventDrivenProgramming) is an object or a HOF or an Eval'd text code snippet or an asteroid behind the scenes would be generally irrelevant to the GUI designer. In fact, it should be mostly declarative so that one is not locked into a specific app language. But that's probably another topic (some of which already exist, such as GuiMarkupProposal). About 95% or higher of GUI actions desired we already know the conventions for such that we can "hard wire" then into a declarative event framework, at least for CBA UI's. We don't need need imperative code for such except the for specialized cases. We mostly have to worry about such low-level issues with today's tool because they suck! They are not abstracting away the common repetitive grunt-work and interaction details. The web made us go backward in time with regard to GUI management, and it also froze the more mature desktop tools, such as VB, Delphi, and PowerBuilder because vendor focus turned to the web in the late 90's. -t

Nice rant, but I'm not sure what it has to do with the real world we work in, except to show you don't like it.

I will concede that HOF's may work better in GUI/clients/browsers based around HOF's. Is that what you want to hear? It seems a narrow victory to me, but if it makes you feel good, then have a badge.

Note that GUI/clients/browsers are merely places where code runs. If HOFs benefit there, they can benefit anywhere code runs.

Yes, being stuck with the damned HtmlStack is indeed "real world", at least in terms of tool choices. But I'm more interested in general patterns of software design of CBA's, not tweaking with or working around flaws in specific languages or browsers. That's help-desk shit.

Are your user-support specialists aware that you speak of them in such disparaging terms? Tsk. Tsk.

Oh, I certainly hope not :-)

By the way, apostrophes indicate possession, not plurality. The plural of "apostrophe" is "apostrophes", not "apostrophe's". Same with "CBAs" and "HOFs"...

There are different opinions on that practice. Acronyms have different rules than "regular" words.

Example: http://public.wsu.edu/~brians/errors/acronyms.html

Well. Fair enough, then.

Even as a generally "normal" programmer, I've found higher-order functions very useful, particularly when doing things like integrating the same internal data objects with external data structures. I can define a function which converts a given data object to an external representation, pass that to a component that does external posts, and separate the concerns beautifully. It's also worth noting that even such simple things as map, filter, and reduce are HOF's. Writing a filter expression in Ruby or lisp is far better than writing 20 lines of code to do the same thing imperatively in Java. -bh

Footnotes

[1] I believe that WetWare issues are very important to software tool choices even though WetWare has been difficult to objectively quantify so far. But at least I cut one some slack if they disagree with a given WetWare model used to judge a tool. I don't feel I can insist something is "better" using WetWare-centric evidence until the science of WetWare is advanced enough (at which point the brain model could take our jobs anyhow :-) BrainEvidenceModelConundrum.

DecemberTwelve

CategoryBusinessDomain CategoryFunctionalProgramming

Loading...