The slow adoption of Functional Programming in banks!
By Abdul Muhit | Oxford Knight
Back in 2012 I wrote a blog covering the rise of functional programming in banks that was picked up by the HackerNews community (The rise of functional programming in Banks). When writing this article I noticed that the use of functional programming (FP), which had minimal presence prior to 2006, was picking up dramatically in banks. My article summed up some of the reasons for the rise in FP in a traditionally OO based environments that are investment banks, and summed up stating FP work in banks would be unavoidable in the future.
Fast forward to now (2014), has the continued rise of FP in banking happened? Is FP being used by all major Investment banks? Is there plenty of work out there for candidates seeking FP work? Are OO developer’s rushing to cross train? Well my answer now in 2014 is……not yet.
Today my job remains almost the as it did last year, and 2006. I am still responsible for placing technologists into bank. As it stands, below is my observation of the FP landscape in banking.
The FP landscape in banks 2014:
- Used at Morgan Stanley across FID and EQ for mostly risk based systems, but also some electronic automated trading
- Trafigura (not a bank but worth a special mention) are using it for risk, pricing, and hedging systems
- Used at RBS alongside Java for a risk aggregation and P&L predict system
- Also, used a small doses at Goldman Sachs, and JP Morgan within a small Structuring technology group, and HSBC for a clearing system and OTC Derivatives, Barclays, and Deutsche Bank in EQD risk engine and Fund Derivatives, and BOAML in Operational technology.
Haskell and F#:
- Used by MAG modelling group at Standard Chartered, by the Global Modelling and Analytics Group (GMAG) at Credit Suisse, and at Barclays within a team focused on Exotic Equity Derivatives.
- Used by Jane Street a prop trading firm in London and NY.
- Is popular within the Credit trading team at Citi and used a tiny bit at UBS on an Equity Derivatives Risk and P&L system.
- Used a fair bit in some gaming shops like Bet365 and telcom companies (Ericsson, Cisco, Alcatel) where its history comes from. Used to be used a tiny bit at Deutche Bank, but I haven’t seen any for a while
The above synopsis may seem like a great deal of work for those interested in FP, but the truth is it’s a very small supply when you look a little bit closer. Let me explain…
- The Haskell and F# work is mostly covered by a very small groups at Credit Suisse, Barclays, and Standard Chartered that consists of a team of well networked and ridiculously bright. There is also a strong bias for perm candidates here over contract.
- The Ocaml work has been used at Jane Street for a long time and they do hire here in London as well as NY. However, Jane Street is known for having an interview process that makes interviews at Facebook, Google, and MemSQL appear like a walk in the park. There is only perm work of offer due to the proprietary strategies being formulated.
- The Clojure team at Citi do hire every so often. However, the volume is probably at best circa 4 a year with a strong bias for perm candidates
- Trafigura and Morgan Stanley have been offering a steady supply of contract and perm Scala work, but the volumes are at best circa 30 a year collectively.
- Scala at JP Morgan, RBS, HSBC, Goldman Sachs Deutsche Bank is collectively a supply of probably 15-30 jobs a year.
Another way of viewing supply/demand is the sheer volume of people in the UK following Scala on Gitbub and Stackoverflow and Meetup. Granted the user base is global and not an exact science, but set against our own manual count the supply of those qualified is not met by the volume of roles available on all major job boards covering IT requirement into finance.
So where is the increased supply I promised last year? I guess in order for me to really answer this question it is probably best to have a quick 2014 synopsis/revisit of what is driving FP in banking:
2014 update: Why are banks adopting FP programming?
1. Concurrent programming and FP: Banks have long used concurrency for performance. However, now with Moore’s Law stalling out processing powers on a single processor has peaked. Servers are now shipped with multiple core and concurrent programming is more of a hot topic in banking industry and outside the banking industry. FP compared to OO programming (say in Java) is better suited with no side effects. FP’s immutability lends itself to concurrent / parallel programming that takes advantage of multi-core and distributed computing. Functional programming’s no shared resource-style lends well to the parallel processing that massive data processing demands.
1. Replacing old Risk Management Systems and productivity: Since 2008, the major banks have embarked on a roadmap for a true enterprise front-to-back, cross-asset, and ‘best of breed’ systems. Reasons for this available here (Abdul Muhit’s answer to Why are banks like JP Morgan and Bank of America Merrill Lynch using Python to replace historic legacy systems built in Java/C++?). Historically, Risk systems at major banks have been built in OO languages (often Java), and replacing these legacy systems is a no easy task. Here FP languages offers the productivity speed needed to get these systems up and running when the need for it was here yesterday! FP languages are more productive because they don’t suffer from the draw-backs of some languages (such as Java). Java for all its success is (according to some FP evangelist I’ve spoken to and some people here > Java Sucks) is a verbose with a lot of boilerplate, realises heavily on IDEs, and code generation to improve productivity. Conversely, advocates of FP claim FP to have a significantly more concise syntax, better support for immutable data structures; better scalability, easier to maintain, which all makes fun! and productive!
2. Industry built upon maths: Banking is a business that is built upon mathematical functions and data transformations. Functional programming is closely modelled on proven mathematics, and of course mathematics is all about abstraction and composition. As such it is these vastly superior abstraction and composition techniques that functional programming give, which allow functional programming teams to cope with the complexity of large codebases and difficult problems.
So to sum up, (1) the need to write better concurrent code, (2) increased productivity amongst FP engineer’s vs OO engineers, and (3) better abstraction and composition is driving FP in banks appear to be the driving force behind adoption. All compelling reasons for pushing a new project in FP in banks, but despite this there are several factors holding banks back.
What’s holding back the adoption of FP in Banks?
In my opinion, there are essentially 3 reason for why banks are slow in adopting FP.
1. Lack of decent tools and support in FP: Back in 2008 when all banks where reviewing their risk systems for a revamp, FP just wasn’t the best option.
- Erlang – is not widely used today, and the tools and support for debugging mathematical problems just wasn’t available in 2008.
- Haskell – mature language with great tools, but the commercial community wasn’t there. Meaning it would be hard to find software engineers, and cross training would take time. Ocaml suffered from the same draw-backs
- JVM FP based languages such as Clojure and Scala provided a decent option for systems historically built in Java, as Java software engineers could understand the syntax. However, the support and tools for debugging just were not available. This has improved dramatically over the past few years with hot-swapping IDE for Scala such as IntelliJ. However, at the time Python tools and support were better and there was no certainty that Scala or Clojure would evolve the way they have.
- F# at the time was also considered as a possible contender, but in the end, unknown to me, banks such as BOAML, JPM, MS opted against it. I suspect it was down them already being invested in the JVM.
Due to these limitation of FP at the time (2008/9), two major banks (namely JPM and BOAML) opted for Python because it had many FP benefits, such as simple grammar, philosophy of clean code, solid standard libraries, good community support, and excellent documentation. Python is a BOAML and JPM investment, and stepping away from it (Despite the Global Interpreter Lock (GIL)) is not going to happen anytime soon no matter how good the benefits of FP offers these days. This essentially means two major banks that employ a lot of London’s software engineers (both contactors and perm) are not going to be hiring FP developers in anger anytime soon.
2. Performance/speed of C++: FP languages such as Haskell can be very fast. In the paper, “Mio: A High-Performance Multicore IO Manager for GHC” (Page on yale.edu), Haskell is shown to outperform C servers.
However, for CPU intensive problems (where time and effort spent is not a factor), it is believed by banking technologists that C++ code written by a talented C++ developer is best suited for high performance systems…say High Frequency Trading (HFT). Therefore, it is this need for speed/performance that will mean C++ will remain dominant over FP in certain areas of finance.
3. Productivity and cost: Many software engineers convincingly argue that FP is far more productive than writing code in OO. However, as it stands hiring FP developers in London more expensive than say a Java developer. A Front Office contractor at Trafigura and HSBC this year are commanding rates of £750 – £800 a day; whilst a Java developer with similar domain knowledge is probably looking at £600 – 700 a day. For there to be a real push towards FP – realised productivity would have to go up a lot. Let’s not forget, most banking systems are there to support the core business of financial trading and the key decision makers are bankers not technologist. At this juncture it’s very difficult to argue to a senior technology manager to justify a complete re-write of an existing system in an expensive language because it was a bit more productive and fun for the engineers.
4. Java 8 (wait and see effect):
Many of the banks have been waiting for Java 8 and its support of Lambda/closures to see if it solves some of the problems associated with Java rather than move towards any FP language or Python. However, this doesn’t appeal to Java/Clojure developers who argue that lack of lambda was never the reason for switching in the first place. Overall, I am not sure what the “let’s wait for Java 8” mentality is all about. We shall wait and see if Java 8 does solve the problems around concurrency, verboseness, and boilerplate code.
What does the future hold for FP in banking?
So to sum up, I still stand by what I said in my blog post last year that there will be no avoiding FP in finance one day. However, shift from OO to FP is not going to happen very quickly despite the spike over the past year. Instead it will be gradual change over the next 5-10 years. The factors that will continue to hold back adoption in the short term are (1) the Python adoption and the passion of the Pythonista’s at BOAML and JP Morgan. (2) the fact that productivity gains in the short term are not high enough to convince senior bankers at boardroom level to part with more money, and (3) the performance benefits of C++ over FP on the HFT side will all mean certain systems are going to be built in C++.
Another thing I see happening in banking over the next 5-10 years is greater range of languages in banking systems (C++, C#, Java, Clojure, Scala, Haskell, F#, Perl etc) than what we have traditionally had. I’ve outlined that Morgan Stanley have gone with Scala and so too have numerous other banks in small doses; however, Clojure is growing fast with an army of Lisp fans switching as I write this. Then another split we might see is from those banks already heavily invested in Microsoft (Credit Suisse, Barclays etc) potentially opting for F#; which will appeal to Microsoft and Ocaml fans.
From a career perspective, it makes sense now to start looking into FP if as an OO software developer you haven’t done so already. At this juncture a strong command of FP is not a prerequisite for the landing a job as a FP Engineer. Some Github, Stackoverflow work would suffice. However, I have noticed Scala requirements becoming more and more detailed over the past 6-12 months; with deep knowledge of Scala API’s, including the collections, as well as performance characteristics of common Scala idioms, Scala Macros and/or Compiler plugins now a prerequisite for some roles. Whilst this may not be seen as a difficult ask at this stage, I expect requirements to become very detailed in a few years. As such, getting your hands dirty now will stand you in good stead.
Get in touch using the contact form below or give us a ring at +44 (0)20 3137 9570 or +1 646 7129 405. We'd love to speak with you about what we can offer.