Oxford Knight Limited

2nd Floor, 33 Queen Street

City of London, EC4R 1AP

T. +44 (0) 20 3137 9570

E. enquire@oxfordknight.co.uk

Oxford Knight

Oxford Knight

You can scroll the shelf using and keys

What is the salary range of Software Engineers in London?

10 July 2014

By Abdul Muhit | Oxford Knight

Money

 

Short answer:

This question is the wrong question to ask if you want to know as an individual what you might be able to earn in London.

Form what I’ve seen London salaries varies dramatically depending on what sector you work, who you work for, your ability, and the nature of your employment contract (perm or consultancy).

Therefore, in order to gauge what you might be able to earn as an individual the question here should be sliced into many sub-questions such as what can I earn at a series A start-up as a ful-stack engineer in London vs SFO? What can I earn as a x10 engineer in London vs SFO? etc.

Nevertheless, my short answer is London pays very well and sometimes more than SFO for a good software engineer (what I mean by “good’ someone who can pass the interview process at Google, Facebook) and a x10 Software Engineer (someone who can pass the interview at companies with a super high bar MemSQL, Palantir, Dropbox, Quora).

Long answer:

Industry:
London (similar to NY) has multiple industry offering in a sense that you have the choice of working for big tech firm Google, Facebook, Twitter etc, some really cool start-ups, or tech-driven funds most of whom are based in and around a 3 mile (5 kilometre) radius. However, the compensation structure for these three environments differs significantly and comparing salary between these different industries is hard let alone against another geographical area such as SFO with a different job market make-up; nevertheless, I shall try.

Cash plus Bonus terms:
Vladimir Novakovski does a great job of outlining how salary structure differ in finance vs tech companies - Career Advice: I just turned 23 and make around $300k working on Wall Street. Could I make this in Silicon Valley if I was looking to make the switch?

In finance it is very normal to be paid a healthy base salary but also gun for a strong bonus based on your performance that can be a multiples of your base cash salary.

Over the past 10 years I’ve seen some jaw dropping bonus earnings by x10 Software Engineers at tech-driven funds that have a hiring bar as high as Jane Street. A few guys I know of have earned north of $1.5m in the HFT space back in 2010. These days those number are not being banished around too often, but it’s still not unheard of to hear a x10 software engineer earning circa £110,000 ($180,000) as a base and £100,000-£200,000 ($165,000 – $330,000) as a bonus. I must stress, these are not bankers I am talking about. These are pure software engineer who code and learnt about finance on the job. They remain hands on and are not necessarily doing anything too quantitative.

Overall it is here in capital markets that I believe a software engineer in London can can earn more than an engineer in  SFO.

From a cash plus stock perspective: 

I am lucky enough to work with a some amazing start-ups working on hard problems that I hope we will disrupt a few sectors in the future. I am also hopeful one day that london we will have its own Facebook equivalent IPO.

However, (I feel like a national traitor saying this) if your looking to make it big on stocks the Bay Area’s past history/recent history, it’s fertile grounds for entrepreneurialism, great VC’s, and baring another Armageddon makes the upside of your stocks a safer bet in SFO than London.

Overall, it is here on the stock option side that I feel that an engineer in SFO is likely to earn more than an engineer in London.

From a pure cash perspective:

For those concerned with pure cash earning (what actually lands in your bank after tax on a month to month basis) I feel that London is on par with SFO if not better placed for a Good or x10 engineer.

Here in London a good — x10 software engineer can expect to earn £100,000 – £130,000 ($165,000 – $215,000) as a software engineer on a perm employee basis working for bank or a tech company.  This does not have to involve being a middle manager, but you may have some technical leadership at the top-end of the salary bracket I’ve outlined.

However, one area that I have to cover that I am not familiar with when it comes to SFO is the consultancy space.

Consultant Software Engineer (those that have 6 – 24 month term) in London (JavaScript, Scala, Java developers) can earn a lot.

I’ve recently dealt with a Java high performance software engineer earning a rate of £1600 ($2600) a day. No domain experience, just an optimization expert. I’ve also seen solid Scala Developer at a rate of £800 ($1300) a day. What makes these London contract rates for the right person so astonishing is that it’s all taxed at a much lower rate of only circa 20%.

So to sum up, for good software engineer (someone who can pass the interview process at Google, Facebook etc.) and a x10 Software Engineer (someone who can pass the interview at companies with super high bar MemSQL/Palantir/Dropbox/Quora) London salaries is higher in cash+bonus terms, maybe pure Cash terms….but may not be in cash/stock terms.

By Abdul Muhit
Linked In: Abdul Muhit – United Kingdom
Email: abdul.muhit@oxfordknight.co.uk
Website: www.oxfordknight.co.uk

 

The slow adoption of Functional Programming in banks!

11 June 2014

By Abdul Muhit | Oxford Knight

main-qimg-5902cd285014f47dbd20ffbe1cfc559b
Back in 2012 I wrote a blog covering the rise of functional programming in banks that was picked up by the HackerNews community (The rise of functional programming in Banks). When writing this article I noticed that the use of functional programming (FP), which had minimal presence prior to 2006, was picking up dramatically in banks. My article summed up some of the reasons for the rise in FP in a traditionally OO based environments that are investment banks, and summed up stating FP work in banks would be unavoidable in the future.

Fast forward to now (2014), has the continued rise of FP in banking happened? Is FP being used by all major Investment banks? Is there plenty of work out there for candidates seeking FP work? Are OO developer’s rushing to cross train? Well my answer now in 2014 is……not yet.

Today my job remains almost the as it did last year, and 2006. I am still responsible for placing  technologists into bank. As it stands, below is my observation of the FP landscape in banking.

The FP landscape in banks 2014:

Scala:

  • Used at Morgan Stanley across FID and EQ for mostly risk based systems, but also some electronic automated trading
  • Trafigura (not a bank but worth a special mention) are using it for risk, pricing, and hedging systems
  • Used at RBS alongside Java for a risk aggregation and P&L predict system
  • Also, used a small doses at Goldman Sachs, and JP Morgan within a small Structuring technology group, and HSBC for a clearing system and OTC Derivatives, Barclays, and Deutsche Bank in EQD risk engine and Fund Derivatives, and BOAML in Operational technology.


Haskell and F#:

  • Used by MAG modelling group at Standard Chartered, by the Global Modelling and Analytics Group (GMAG) at Credit Suisse, and at Barclays within a team focused on Exotic Equity Derivatives.

OCaml:

  • Used by Jane Street a prop trading firm in London and NY.


Clojure: 

  • Is popular within the Credit trading team at Citi and used a tiny bit at UBS on an Equity Derivatives Risk and P&L system.


Erlang:

  • Used a fair bit in some gaming shops like Bet365 and telcom companies (Ericsson, Cisco, Alcatel) where its history comes from. Used to be used a tiny bit at Deutche Bank, but I haven’t seen any for a while

The above synopsis may seem like a great deal of work for those interested in FP, but the truth is it’s a very small supply when you look a little bit closer. Let me explain…

  • The Haskell and F# work is mostly covered by a very small groups at Credit Suisse, Barclays, and Standard Chartered that consists of a team of well networked and ridiculously bright. There is also a strong bias for perm candidates here over contract.
  • The Ocaml work has been used at Jane Street for a long time and they do hire here in London as well as NY. However, Jane Street is known for having an interview process that makes interviews at Facebook, Google, and MemSQL appear like a walk in the park. There is only perm work of offer due to the proprietary strategies being formulated.
  • The Clojure team at Citi do hire every so often. However, the volume is probably at best circa 4 a year with a strong bias for perm candidates
  • Trafigura and Morgan Stanley have been offering a steady supply of contract and perm Scala work, but the volumes are at best circa 30 a year collectively.
  • Scala at JP Morgan, RBS, HSBC, Goldman Sachs Deutsche Bank is collectively a supply of probably 15-30 jobs a year.

Another way of viewing supply/demand is the sheer volume of people in the UK following Scala on Gitbub and Stackoverflow and Meetup. Granted the user base is global and not an exact science, but set against our own manual count the supply of those qualified is not met by the volume of roles available on all major job boards covering IT requirement into finance.

So where is the increased supply I promised last year? I guess in order for me to really answer this question it is probably best to have a quick 2014 synopsis/revisit of what is driving FP in banking:

2014 update: Why are banks adopting FP programming?

1.    Concurrent programming and FP: Banks have long used concurrency for performance. However, now with Moore’s Law stalling out processing powers on a single processor has peaked. Servers are now shipped with multiple core and concurrent programming is more of a hot topic in banking industry and outside the banking industry. FP compared to OO programming (say in Java) is better suited with no side effects. FP’s immutability lends itself to concurrent / parallel programming that takes advantage of multi-core and distributed computing. Functional programming’s no shared resource-style lends well to the parallel processing that massive data processing demands.

1.   Replacing old Risk Management Systems and productivitySince 2008, the major banks have embarked on a roadmap for a true enterprise front-to-back, cross-asset, and ‘best of breed’ systems. Reasons for this available here (Abdul Muhit’s answer to Why are banks like JP Morgan and Bank of America Merrill Lynch using Python to replace historic legacy systems built in Java/C++?). Historically, Risk systems at major banks have been built in OO languages (often Java), and replacing these legacy systems is a no easy task. Here FP languages offers the productivity speed needed to get these systems up and running when the need for it was here yesterday! FP languages are more productive because they don’t suffer from the draw-backs of some languages (such as Java). Java for all its success is (according to some FP evangelist I’ve spoken to and some people here > Java Sucks) is a verbose with a lot of boilerplate, realises heavily on IDEs, and code generation to improve productivity. Conversely, advocates of FP claim FP to have a significantly more concise syntax, better support for immutable data structures; better scalability, easier to maintain, which all makes fun! and productive!

2.    Industry built upon maths: Banking is a business that is built upon mathematical functions and data transformations. Functional programming is closely modelled on proven mathematics, and of course mathematics is all about abstraction and composition. As such it is these vastly superior abstraction and composition techniques that functional programming give, which allow functional programming teams to cope with the complexity of large codebases and difficult problems.

So to sum up, (1) the need to write better concurrent code, (2) increased productivity amongst FP engineer’s vs OO engineers, and (3) better abstraction and composition is driving FP in banks appear to be the driving force behind adoption. All compelling reasons for pushing a new project in FP in banks, but despite this there are several factors holding banks back.

What’s holding back the adoption of FP in Banks?

In my opinion, there are essentially 3 reason for why banks are slow in adopting  FP.

1.    Lack of decent tools and support in FP: Back in 2008 when all banks where reviewing their risk systems for a revamp, FP just wasn’t the best option.

  • Erlang – is not widely used today, and the tools and support for debugging mathematical problems just wasn’t available in 2008.
  • Haskell – mature language with great tools, but the commercial community wasn’t there. Meaning it would be hard to find software engineers, and cross training would take time. Ocaml suffered from the same draw-backs
  • JVM FP based languages such as Clojure and Scala provided a decent option for systems historically built in Java, as Java software engineers could understand the syntax. However, the support and tools for debugging just were not available. This has improved dramatically over the past few years with hot-swapping IDE for Scala such as IntelliJ. However, at the time Python tools and support were better and there was no certainty that Scala or Clojure would evolve the way they have.
  • F# at the time was also considered as a possible contender, but in the end, unknown to me, banks such as BOAML, JPM, MS opted against it. I suspect it was down them already being invested in the JVM.

Due to these limitation of FP at the time (2008/9), two major banks (namely JPM and BOAML) opted for Python because it had many FP benefits, such as simple grammar, philosophy of clean code, solid standard libraries, good community support, and excellent documentation. Python is a BOAML and JPM investment, and stepping away from it (Despite the Global Interpreter Lock (GIL)) is not going to happen anytime soon no matter how good the benefits of FP offers these days. This essentially means two major banks that employ a lot of London’s software engineers (both contactors and perm) are not going to be hiring FP developers in anger anytime soon.

2.    Performance/speed of C++: FP languages such as Haskell can be very fast. In the paper, “Mio: A High-Performance Multicore IO Manager for GHC” (Page on yale.edu), Haskell is shown to outperform C servers.

However, for CPU intensive problems (where time and effort spent is not a factor), it is believed by banking technologists that C++ code written by a talented C++ developer is best suited for high performance systems…say High Frequency Trading (HFT). Therefore, it is this need for speed/performance that will mean C++ will remain dominant over FP in certain areas of finance.

3.   Productivity and cost: Many software engineers convincingly argue that FP is far more productive than writing code in OO. However, as it stands hiring FP developers in London more expensive than say a Java developer. A Front Office contractor at Trafigura and HSBC this year are commanding rates of £750 – £800 a day; whilst a Java developer with similar domain knowledge is probably looking at £600 – 700 a day. For there to be a real push towards FP – realised productivity would have to go up a lot. Let’s not forget, most banking systems are there to support the core business of financial trading and the key decision makers are bankers not technologist. At this juncture it’s very difficult to argue to a senior technology manager to justify a complete re-write of an existing system in an expensive language because it was a bit more productive and fun for the engineers.

4.   Java 8 (wait and see effect): 
Many of the banks have been waiting for Java 8 and its support of Lambda/closures to see if it solves some of the problems associated with Java rather than move towards any FP language or Python. However, this doesn’t appeal to Java/Clojure developers who argue that lack of lambda was never the reason for switching in the first place. Overall, I am not sure what the “let’s wait for Java 8” mentality is all about. We shall wait and see if Java 8 does solve the problems around concurrency, verboseness, and boilerplate code.

What does the future hold for FP in banking?

So to sum up, I still stand by what I said in my blog post last year that there will be no avoiding FP in finance one day. However, shift from OO to FP is not going to happen very quickly despite the spike over the past year. Instead it will be gradual change over the next 5-10 years. The factors that will continue to hold back adoption in the short term are (1) the Python adoption and the passion of the Pythonista’s at BOAML and JP Morgan. (2) the fact that productivity gains in the short term are not high enough to convince senior bankers at boardroom level to part with more money, and (3) the performance benefits of C++ over FP on the HFT side will all mean certain systems are going to be built in C++.

Another thing I see happening in banking over the next 5-10 years is greater range of languages in banking systems (C++, C#, Java, Clojure, Scala, Haskell, F#, Perl etc) than what we have traditionally had. I’ve outlined that Morgan Stanley have gone with Scala and so too have numerous other banks in small doses; however, Clojure is growing fast with an army of Lisp fans switching as I write this. Then another split we might see is from those banks already heavily invested in Microsoft (Credit Suisse, Barclays etc) potentially opting for F#; which will appeal to Microsoft and Ocaml fans.

From a career perspective, it makes sense now to start looking into FP if as an OO software developer you haven’t done so already. At this juncture a strong command of FP is not a prerequisite for the landing a job as a FP Engineer. Some Github, Stackoverflow work would suffice. However, I have noticed Scala requirements becoming more and more detailed over the past 6-12 months; with deep knowledge of Scala API’s, including the collections, as well as performance characteristics of common Scala idioms, Scala Macros and/or Compiler plugins now a prerequisite for some roles. Whilst this may not be seen as a difficult ask at this stage, I expect requirements to become very detailed in a few years. As such, getting your hands dirty now will stand you in good stead.

By Abdul Muhit
Linked In: Abdul Muhit – United Kingdom
Email: abdul.muhit@oxfordknight.co.uk
Website: www.oxfordknight.co.uk

How to Resign: Some Tips for a Smooth Transition

1 May 2014

By Alex Betteridge | Oxford Knight

alex

Resigning is a huge career decision whether it’s from your first job out of University or your final move as an executive at the top of your game. There is a huge amount to consider and if you’re serious about it there are certain dos and don’ts to make the process as smooth as possible. Resigning from your job doesn’t have to mean the only positive is moving into a new shiny job – but if you’re not careful it can be an ugly and often awkward process. Here are some tips and hoops you should jump through when going through the resignation process that should help omit that common occurrence of a sour departure!

If you’re looking to move, prepare to leave!

From my experience as a tech recruiter it’s very easy for techies to peruse the market, check that they aren’t missing out on superior opportunities and of course clarify that they aren’t being underpaid/undervalued in their current role. However it’s another ball game altogether when it comes to actually leaving. If you have started interviewing elsewhere then you should synonymously begin to prepare and understand the logistics of leaving your current employment.

Make sure you:

  • Check your notice period and whether it’s negotiable.
  • Check if you have fulfilled your probation period.
  • Check if there is a handing over clause which states an amount of time you must remain at the company to ensure the repossession of your role.
  • Check you have pursued all avenues of advancement in your company, providing you aren’t completely dissatisfied with your company altogether!
  • Think about the implications of you leaving not just in terms of career and compensation but also relating to location, family and relationships built up in your current role.

Still going to resign? – Don’t be disruptive and respect your boss.

If your boss isn’t the first to know about your resignation, it probably won’t go well. Telling your boss before anyone else matters because:

  • They will value the fact you haven’t disrupted others in the workplace with your uncertainty. A decision to move on might plant an infectious seed that could lead to others considering resignation.
  • You may begin the process of resignation and switching companies but your offer elsewhere may fall through. This is why it’s important to break the news to fellow colleagues only once your move is set in stone.
  • Keeping your boss out of the loop will irrefutably affect your reference if you do leave and potentially tarnish all those years of hard work built up on your CV.

Initially breaking the news that you have decided to leave will vary depending on the relationship with your boss. Ideally it’s a conversation that can be had amicably as well as professionally. Be polite, to the point and don’t let your bosses reaction affect your decision last minute.

Prepare your resignation in written form AND orally:

Handing in your written notice is one thing but prepare some words in your head too! Your boss will react (unless he’s about to fire you anyway!), so it’s important to stick to what you’ve planned to say, make it clear you are submitting an oral resignation and be confident in your decision. If you’ve thought it through thoroughly, you shouldn’t have any hesitations. After resigning orally, be sure to leave the meeting on a positive note and make clear that you’ve valued your time there very highly (even if you haven’t).

The Counter Offer:

This is a common spanner in the “job switching” works, especially for a recruiter! The fact you are ready to leave your company probably already rules out considering a counter offer but until it’s in front of you (make sure it’s written and signed by the powers that be), you never quite know how you might react. Make sure you don’t give away any inclination of interest in a counter but ALWAYS go away after receiving one and think through it pragmatically:

  • What are you ACTUALLY being offered? Make sure it’s not just a new title.
  • Don’t just look at what precedes the 000’s. Is the “new” role a constructive career move/better than what else is on the table?
  • Will the fact you have voiced wanting to leave the company mean fulfilling any future role there is unrealistic?
  • Your boss may now be doubtful of your 100% commitment to the firm. In this way, maybe it would be better to move on.
  • Google it! There are plenty of stories shared and studies exhibited on the net about counter offers and how they played out in varying circumstances. You’ll find there is compelling evidence to steer well clear of them on the whole!
  • A counter is more often than not a reactive tactic. Replacing an employee costs time and money so be sure to see through the sudden “loving” of you like they’ve never “loved” you before drivel.
  • If you at any point say; “I will listen to what you may have to offer” then you risk burning your bridges on both sides. To elaborate on that, if your boss invests time and effort in building a counter offer and you then reject it, this will add insult to injury. Similarly by contemplating a counter which can take some time to be drawn up, you risk aggravating your future employer by keeping him in the dark.
  • Always be appreciative of a counter, if you receive one then you know you are valued by them and it bodes well for receiving a nice reference and a compassionate departure.

In my opinion if you are thinking about resignation it is generally a positive thing. By exercising your right to leave and bite the hand that holds your lead you will leave whatever it is that you feel is stagnating your career progression. Conversely by remaining where you are (accepting a counter) you live with the anxiety of “what could have been” and will invariably lose focus and deteriorate in performance. Moreover extensive research in counter offer acceptance has been carried out showing just 6% of counter takers remaining at their company after 12 months.

Resignation isn’t a negative word!

I think a good place to conclude is in reflection of my own experiences as a recruiter.  That may predispose the value of this blog to many but I’m a big believer in decisions based on hard evidence. My recruitment days are not extensive (I’m still in my first year), but that’s not to say I haven’t already seen and learnt a huge amount. From my placements thus far I’ve seen the importance of jumping through the hoops detailed above. If you’re respectful in the way you handle switching jobs you can’t go far wrong. Resignation is not a negative thing, there’s a little known thing called the circle of life – someone will replace you and you will replace others! Just be professional, know what you want and stick to your guns!

 

Why are banks like JP Morgan and Bank of America Merrill Lynch using Python?

30 April 2014

by Abdul Muhit | Oxford Knight

Python1

 

I am not technical and don’t write code (yet), so I’ll leave it to others to cover the low level technical details for Python’s adoption.

From a high level perspective, the decision to go with Python to rewrite core systems at firms such as BOAML and JP Morgan is to build a more transparent and collaborative system in firms that have people of varying technical ability and who’s chief role is not always to write software.

The 2008 financial crisis will be remembered as the year of terrible events at corporate and systemic level. Banks up until this point failed due to a lack of collaboration.

Despite what might seem as a logical step to those in the tech industry of building one Risk management system across all individual business areas and so creating a greater level of transparency and leveraging economies of scale, banks continued until 2008 to build individual risk management systems that didn’t talk to each other. Various reasons for this, but essentially it prevented collaboration.

Since 2008, the major banks have embarked on a roadmap for a true enterprise front-to-back, cross-asset, ‘best of breed’ systems. This is no longer the sole agenda of technology teams but is also necessitated at boardroom level and demanded from external institutions that look for transparency for us, the tax payer.

Since early 2009, JP Morgan have been rolling out the Athena Platform (a Python along with C++ and Java for speed), a cross-market risk management and trading system.

BOAML after the merger between BOA and ML also decided at management level to also build a similar system to Athena called Quartz.
At this juncture, it’s JP Morgan and BOAML that are the two banks using Python in anger.

Ironically, or maybe deliberately as there is some legal issues around this: both senior managers spearheading the move to cross asset risk at JP Morgan and BAOML had initially worked at Goldman Sachs.

As a bank, Goldman Sach’s proprietary risk management platform built using SecDB/Slang has been seen in the eyes of many as having done a better job in comparison to its competitors. SecDB is a GS prop database and Slang is a lightweight scripting language similar to Python/Perl with the ORM layer baked in.

Not much is known of Slang/SecDB because it’s a prop language/DB, but one of the major advantages noted of Slang (like Python) is its lightweight language that anyone can get up to speed with in a short space of time. Also, given that Slang is used everywhere at GS front/middle/back office, it allowed 15 years ago something multiple systems at other banks built using multiple languages couldn’t do – collaboration. It may seem trivial these days, but GS as a bank could, with ease, do database replication allowing changes in London to be seen in Tokyo almost instantly.

Just like Slang, Python is a simpler (ish) language than C++/Java; which matters when you have people that don’t have a strong grasp of software engineering. Hence, when choosing to build a true enterprise front-to-bank cross asset risk systems for the new financial era, JPM and BOAML have opted for Python.

 

By Abdul Muhit

Linked In: http://uk.linkedin.com/in/abdulmuhit

Email: abdul.muhit@oxfordknight.co.uk

Website: www.oxfordknight.co.uk

Oxford Knight is a technical recruitment agency. None of our consultants have written a line of code… yet. We apologise if this article doesn’t keep some purist happy, but we’re trying to build a new generation of technical recruitment agencies…. We listen, participate, and deliver.

 

“OMG Drones” The London Clojure Exchange

19 December 2013

By Ben Rogan | Oxford Knight

Last week I went along to the LCE (London Clojure Exchange), not to be confused with the London Camera Exchange or the Airport in Honduras. Laden with the Black Death / SmallPox, I wasn’t at the top of my game, so if any of London’s Clojurians are sick, I can only apologise. We can at least refer to #ClojureX as ground zero for the end of the world.

*Edit* Bubonic plague discovered in Madagascar – not my fault.

The Quick Overview:

The London Clojure Exchange, hosted by SkillsMatter is meant to be a collection of some of the most exciting and interesting ideas the London Clojurians and @Otftom can put together.

For the Non-Technical/Uninitiated:

Clojure is a dialect/sub language of the Lisp variety. Originally created by Rich Hickey, Clojure is a general FP language that runs on the JVM, CLR and JS engines, it compiles directly to JVM bytecode, yet remains completely dynamic as every feature supported by Clojure is supported at runtime!

Clojure has a strong focus on programming with immutable values (although there is a little mutable state) and is particularly good for developing multithreaded systems. It hasn’t grown with the academic rigor of Scala per se, but it has been able to incorporate some of the best elements of a number of languages. In addition to this, because Clojure is a dialect of Lisp and shares with Lisp the code-as-data philosophy, it has a powerful macro system to boot.

For those who haven’t tried Clojure, and want to give it a try, take a look at: http://tryclj.com/ or attack some problems at http://www.4clojure.com/problems.

The Clojure Exchange:

Learning to talk to machines with speech acts – a shipwreck adventure, by Carin Meyer

As a good author, I caught you with the title. Now I can explain. The LCE was opened by Carin Meyer:http://gigasquidsoftware.com/  flying a Parrot AR drone around the room to save her from the theoretical desert island she is stranded on.

The talk was awesome, as was the drone, especially as it lacked the venom of the USAF, and it bobbed happily to some music later in the day as well.  Carin’s talk covered speech acts, the philosophy of language and building a language with Clojure and Instaparse, not to mention some McCarthy AI papers, which managed to captivate the crowd.

Credit – Photo by Anna Pawlicka

You can watch the lecture here: http://skillsmatter.com/podcast/home/carin-meier but to be honest. Drones. Come on people. Do I need to sell it any more

Rob Rees, on Reiman

Rob followed Carin, not an easy task. Especially as he lacked drones. Rob is one of the lead developers at the Guardian and has been pushing the Clojure Dojos along at Thoughtworks, he actually helped me crack my first few lines of code. Rob gave an interesting talk on Reiman and complex event processing.

Rob gave us the run down on how Riemann works and why it might be a good idea to rub your metrics through it, and ran us through writing a stream processor. For those who aren’t familiar, Riemann aggregates events from your servers and applications with a powerful stream processing language.

Scala Vs Clojure, by David Pollak

Just a quick FYI on this section – David and I are bound to cause some controversy, Clojure Vs Scala. We all know the obvious choice is *coughs* but here is a quick summary of what happened, and the debate at the moment.

Dave was, and remains to a certain extent a big contributor to the Scala World, I think he is still one of the primary contributors to Lift, something he gave a little talk about at ScalaX earlier in the week.

Lift itself has caused some controversy, the security, ease of use and philosophy have had positive feedback. Lift competes pretty heavily with Play on the web framework front, and from a recruitment perspective, places like Morgan Stanley, Various Gaming Companies or Trafigura tend to look for Play over Lift; I believe FourSquare are using lift at the moment, but I haven’t seen it anywhere else, as of yet.

David has some pretty interesting thoughts on Scala, and he has been using it professionally at Brick Alloy for a while now.

Scala, has a more academic background, as a result has good documentation. Odersky is after all an academic, so publishing has been driving Scala’s growth. It was initially designed as a better Java, with stronger theoretical underpinnings, and designed by guys who are supposedly smarter.

The tooling in Scala leaves a lot to be desired, Eclipse has spent the last 7 years not working http://scala-ide.org/docs/current-user-doc/faq/known-issues.html, but Intellij is pretty good (now, and if you have a 2gb heap). If you were going to take an aggressive look at Scala, a beautiful language, there are those who would argue that there is extensive version fragility and that the language and the library are instable and problems in pattern matching data-structures have only just been remedied, not to mention the compile time…

All that being said, these are the thoughts of someone who still uses Scala professionally, and runs their own Scala consultancy and recognises that Scala is still great for refactoring codebases and viewing complex data.

On the flip side to all this ranty-ness, are some pretty exciting times for Clojure. It is a community that has a certain maturity about it and isn’t weighed down by the academic rigour.  The academic rigour which, as I can say after two degrees, doesn’t necessarily always help.

Clojure is great for shipping, and on top of that you can work in the REPL, building or growing the program line by line, leaf by leaf, “gluing tiny bits of excellence together”. Clojure has been very reactive to other languages, and with a community of polyglots, hasn’t put itself above stealing the best bits of other languages, meaning it is simple, easy to read and understandable, as the ideas are intentionally less complex. But there are a lot of parentheses.

This is a bit biased, I am learning Clojure and feel a little affinity for it, but there is some truth to the statement that the libraries just work, it is relatively stable – ish and the REPL development is excellent.

*I’d be keen to hear some feedback on this section, it is leaning towards the more technical of the things I have written recently, and I don’t want to step on any toes! But I don’t want to pull any punches either. Good writing is good opinionated research.

Mining Social Data with Cascalog, Tom O’Brien

Mining twitter seems to be a pretty interesting trend at the moment, there is a lot you can do with the data, irrespective of the language behind it. You only have to look at the recent purchase of Topsy:http://www.theguardian.com/technology/2013/dec/02/apple-buys-topsy-twitter-analytics-report-200m to see exactly what is happening.

You can find a pretty rudimentary (but fun sentiment analysis tool here: http://www.sentiment140.com/search?query=jackwills&hl=en)

Tom gave an interesting talk on using Cascalog2 to determine groups of similar twitter users, which probably have some overlap with the day job at Likely Ltd. FYI, Cassalog is a Clojure library for writing Hadoop jobs using a declarative syntax. If you have 40 minutes spare – take a peek at the lecture here:http://skillsmatter.com/podcast/home/mining-social-data-with-cascalog

Sam Aaron – Algo Raves & Overtones

Sam gave us another, completely different talk on Clojure, and programming in general. He opened by positing some pretty interesting questions including; what would coding be if it wasn’t classified as Engineering? Could it be an Art? Especially if you think, like with Clojure, you take a program and grow it, rather than fit the pieces together. He also pointed out that the general perception of software development really is so perverted.

Sam was trying to argue that coding can be considered a form of expression, beyond some cheap hacking shot on Saturday night TV.

Then he set out to shatter some illusions, trying to communicate to us what coding really could be. When he wasn’t trying to fall off the stage, putting together some live coding, turning code to music, I can’t really give you an idea of what it sounds like, (I have the musical talent of a gofer) but I’d recommend you take a look at a few of the links below:

Making Music with Overtone: http://jvns.ca/blog/2013/11/19/day-30-making-music-in-clojure-with-overtone/. You can find a pretty simple overtone tutorial here: http://vimeo.com/22798433

Overtone: http://overtone.github.io/ & Algo Rave: http://algorave.com/

Summary:

The London Clojure community might still be small, but as we saw from ScalaX, it doesn’t exactly take long for a language to take off. Just look at the amount of funding Typesafe have! And the language is already being picked up by some key players, Banks, Hedge Funds, Consultancies, Big Data Analytics and Social Media and all getting their hands dirty with Clojure.

I’d love to hear your thoughts, especially on Drones or Overtone, but anything Clojure/Tech related is welcome. And just because its it is nearly Christmas; take a look at this: http://www.kaggle.com/c/packing-santas-sleigh

So, at this point I’ve run out of things to say, so I’ll sign off. Always open to opinions – constructive and polite, I’m still a relative beginner, to both Clojure and Scala so I’d love to hear your thoughts.

 

 

 

The Scala Exchange #Scalax

4 December 2013

By Ben Rogan | Oxford Knight

So, it’s Monday at 9am. I’m at The Scala Exchange (ScalaX) run by Skills Matter, and I’m preparing for 48 hours of Scala.

The idea? That hundreds of the most talented and eager developers from the Scala community, mingle, swap ideas and solutions over coffee and lectures. And pick up some free T-Shirts. I’m all about the free T-Shirts.

There are lots of recruiters out there who really don’t know the first thing about the industry they recruit for, sometimes it’s a wonder their knuckles don’t drag along the floor. At Oxford Knight we are given a free reign to learn whatever we can, learn to code, go to events, and source candidates in ways more effective than a cold call, so this year, Phoebe, Sean and I popped along.

These events are an amazing way to, pick up a thing or two about the Scala Community in London, see what companies are using Scala in anger and how Akka, Spray, Lift and Scala in general is doing.

Scala – Quick Summary (for the non technical)

  • I doubt anyone reading this won’t know what Scala is; but just in case, Scala is a rapidly growing programming language, it is statically typed and lightweight, and picking up a lot of speed in London and the US. Right now it is being used in Hedge Funds, Banks, Big Data and Research, so it’s quite an exciting time for the community in general.

The central tools for Scala are Akka, Lift and Play, the quick breakdown is:

  • Akka is a toolkit and runtime system designed for building highly concurrent, fault tolerant systems. It is particularly good for scaling out, just join the cluster! And the concurrency model is clear and logical (Mostly).
  • Lift is a web application framework that is designed for Scala, it seems to have some great security features especially resistant to OWASP vulnerabilities. It also has the benefit of being able to handle huge traffic levels.
  • Play is a lightweight web framewok that is picking up speed. It is being used commercially at the Guardian now and play2 if fully RESTful.
  • Spray is an open-source toolkit for building REST/HTTP-based integration layers on top of Scala and Akka.Check it out here: http://spray.io/scala.io/#/

The Exchange

Anyway, I have my badge, ‘Ben Rogan, Oxford Knight’ suitably ambiguous, and I walk into our first lecture. Eager as a kid at Christmas to get stuck in.

Lenses, by Simon Jones

So, I know you are rapt in your seats. What was the first lecture? Intermediate Level Haskell, by Simon Jones. That’s a bit of a curveball especially at 9am. Well. Despite the obvious left field choice for the opening talk at ScalaX. Simon gave us all the run down on how to implement Lenses on Data Structures in Haskell, albeit trying to convert some of the crowd.

He might have had more success were it not a level of enthusiasm that was incomparable with Monday Mornings. Nonetheless, an interesting and energetic talk to begin the conference. And I definitely need a cuppa after that one. You can find the online lecture here: http://skillsmatter.com/podcast/scala/simon-peyton-jones and there is a more Haskell friendly lecture here: http://www.youtube.com/watch?v=cefnmjtAolY

Failure. Deal With it, by Victor Klang

I won’t delve into this one too much, more to the point watch it here: http://skillsmatter.com/podcast/home/viktor-klang it was a great talk about everything that can go wrong with your system from hardware, though to design flaws and the idiots you work with. Comedy points of the talk go to Victor, there were both jokes and laughs.

Victor gave a great run down on why Knight Capital took a pre-tax loss of $440m on August 1st, just because someone didn’t update the servers. “Someone was very sad that day” as Victor put it.

Implicit Conversions => Implicit Parameters, by Bill Venners

Bill (Who I met. High five for me) gave an interesting keynote, demo-ing the tradeoffs between implicit conversions and parameters, running a lot of demos in the Scala REPL, that I’ll admit lost me a bit.

The Road to Akka Cluster, and Beyond, by Jonas Boner

Jonas gave an interesting talk on day two, at the keynote. You can find the lecture here: http://skillsmatter.com/podcast/scala/jonas-boner he also had some interesting views on the CAP Theorem: http://en.wikipedia.org/wiki/CAP_theorem in particular how it had a narrow focus and how it ignores latency issues.

He also made me google atomicity in database systems. It might be worth updating yourself at: http://en.wikipedia.org/wiki/Atomicity_(database_systems)

 

Moody Tweets, by Jan Macháček

Perhaps my favourite talk of the weekend #MoodysTweets, Jan gave an exciting and fast paced live coding session building a mood analysis tool using Scala and Spray (Very different to my attempt in Clojure)

A pretty cool lecture all in all, we got to code along putting together the solution from Jan’s Github, on our laptops (well everyone else was using a Mac *hints at boss* and some guidance on putting the API together. We were actually replicating what it took some researchers at Loughborough four Months to put together. Fun and Exciting all round, while the compile time wasn’t great, and I hit a fair few typos, Jan put together an interesting, working prototype in 40 Minutes! The fact that sentiment analysis is the kind of work that is growing rapidly, especially when looks at through the infamous big data eye (I just ran a few maps), look at Datasift….

Some nice working examples can be found here: http://blog.mashape.com/post/48757031167/list-of-20-sentiment-analysis-apis the lecture is here: http://skillsmatter.com/podcast/scala/moody-tweets

Clojure and Visualisation, by Neale Swinnerton

Neale from Mastodon, those data visualisation guys who work mostly in clojure, came and gave us a Clojure taster session. He built a data visualisation tool, mapping followers to contributors on github. Lots of playing in the REPL and a few converts. Not too much else to report here but if you are looking for reasons to try clojure its only 45 minutes! http://skillsmatter.com/podcast/home/neal-swinnerton

Summary:

This was a great event, I would heartily recommend, albeit there is quite a price tag. The speakers were top notch, even if some of the topics were off piste and a tad tricky to follow. I fundamentally regret every beer I drank with the Typesafe and Guardian Guys and now really want a Mac. Apart from that – give it a try. (Or watch them online for free…but it isn’t the same).

Any thoughts or comments. Get in touch:

 

All That Jazz…

19 September 2013

By Phoebe Kemp | Oxford Knight

Sheet-Music

The last good thing written in C was Franz Schubert’s Symphony Number 9.                     – Erwin Dieterich, programmer

As an IT recruiter I see many software engineering profiles. And although I do not code myself (yet…), there is often something I have in common with developers: a passion for music and, more specifically, jazz.

Is it just coincidence that so many developers are also jazz musicians – or are there some profound similarities between the two?

A bit of research and philosophising later, here are some thoughts.

To begin with a general statement: the history of coding is similar to that of jazz.

Nick Sieger makes an argument for the similarities between the history of jazz and programming in a blog article from 2008. In his own words:

‘You could break it down like this: C is akin to New Orleans “hot” jazz: foundation for much that followed, and still relevant in a lot of ways, but definitely showing its age. Java is like swing (literally! c.f javax.swing.*): appeals to the masses, anyone gets it without a lot of cognitive load. Ruby is like bebop: favors minimalism, more powerful, more intellectual, and less understood.

So what does that mean for the future of programming? We’re already seeing an increase in DSLs, polyglotism, and languages and tools used for specialized purposes. I would expect that trend to continue, and using the development of the many styles of jazz as a ruler, would say that there will not be a Next Big Language (NBL). Specialization and niche skills will be increasingly important as differentiators in our careers, but that means there will also be plenty of opportunity to stake out that new ground for ourselves and become leaders in these new specializations.’

Taking this argument further, we might view the JVM based functional programming languages as a kind of improvisation on the 12 bar blues – both database and 12 bar blues are pretty low risk forms, able to support indefinite and complex patterns on top.

But coincidences and nice analogies aside, do the practises of coding and jazz have anything more in common?

Music theory has the same elegance and functionality shared by coding

In a crude nutshell, programs need the building blocks of algorithms, patterns, frameworks and libraries – and they need these building blocks to be clear and unambiguous.

Jazz also has certain fundamentals that are key – pun intended – to any piece. A band that hasn’t coordinated aspects like rhythm and chord progression is not going to get very far. Think of a saxophonist playing Take Five in 4/4, or Chick Corea’s Spain with the wrong harmonic progression. It just wouldn’t work.

Some inventive developers have actually tried to mix coding and jazz with the programming language Velatio, using MIDI files as source code. The programs are defined by pitch and order of notes and this, interestingly, leads to jazz-like harmonies. Here is a sheet music representation of a “Hello, World” program:

Beyond the structure, coding and jazz are also open-ended and allow minds to be creative.

In a talk on Big Data, Stephen Cohen makes it clear that the quantitative aspects of Data – the building blocks – are necessary but they can’t do everything (http://www.youtube.com/watch?v=VVw2RMun-ss). Algorithms can’t process qualitative data or capture subtle context. That’s why, according to Cohen, humans will always be necessary in dealing with computers. Human qualities cannot be quantified.

The balance of analytics and emotions is also vital in jazz. No matter how complex the rhythm and chord patterns, it’s the human element of improvisation and tone that distinguish the best jazz. And often, the simpler the better: have a listen to Miles Davis’ long notes and pauses in ‘Round Midnight, http://www.youtube.com/watch?v=td3SE3zEVP0.

So the listener takes the building blocks as a given, allowing them to focus on the human touch, emotion and application. Just so in programming.

Programming = Performance?

Programming blogger Chad Fowler recently compared programming to performance. He argues that programming is often, mistakenly, focused on the end product, where as in actual fact development ‘is itself a process’:

The best development happens not when programmers are keying in the code to implement a design but when great programmers are improvising within constraints and iterating toward a goal.

So the programmer, like the solo improviser, works on intricacies of pattern against more human qualities of tone and emotion.

If you’re interested in delving into the deeper similarities, Rich Hickey explores the nature of design and composition in software and music in this talk (http://www.infoq.com/presentations/Design-Composition-Performance), mentioning that the act of improvisation is not something spontaneous but something that takes extensive study and practise.

Hickey then combines the similarities of programming and performance in a neat summary:

Design like Bartok

Code like Coltrane

If you have any thoughts or experiences on combining jazz and coding, I’d been keen to chat to you.

Oxford Knight is a technical recruitment agency. None of our consultants have written a line of code… yet. We apologise if this article doesn’t keep some purist happy, but we’re trying to build a new generation of technical recruitment agencies…. We listen, participate, and deliver.

 

 

How to deal with difficult recruiters…

10 July 2013

By Zahra Al-Nugaidi | Oxford Knight

4524246185

 

We’ve all been there and we’ve all dealt with one; trying to shake off a shoddy recruiter is something that we all have to face in our careers at some point.

There are a number of myths and misconceptions about how recruiters should and do work and even why their job function exists nowadays. Unfortunately, this means that many job seekers out there don’t realise how to utilise them as a valuable resource.

Like all professions, there are good recruiters and bad recruiters. I would be lying if I said there weren’t a number of recruiters out there who don’t lie, manipulate, aggressively ‘persuade’ and promise things they can’t deliver on and unfortunately the sheer number of these types have meant that often the entire profession gets given a bad name. In this article, I’m hopefully going to highlight traits of these types and show you how to avoid them, or if they’ve already put you in the process with a company you really like, how to deal with them…

  • CV spraying… One of the first questions I usually get asked when engaging with a new candidate is “Will you send my CV out without my permission?”. If this has happened to you then, first of all, well done! You’re clearly a great candidate whose profile is so perfect that the overzealous recruiter has got excited and tried to cover all his bases. Unfortunately for you, it’s a pretty frustrating situation, particularly if you’re dealing with multiple agencies who might be working with the same client. My suggestion is to spell it out to the recruiter clearly from day 1…”Hey recruiter Bob, if you send my CV out without a permission email from myself, I will never use your services again. Thanks!” This should give them the right idea and hopefully scare them into best practice if not…
  • Sell sell sell… I’m often astounded at the level of which many recruiters dress up the jobs they are recruiting for. At the risk of sounding like your mother, if something sounds too good to be true, it probably is. That’s not to say that there aren’t some amazing jobs out there but if a recruiter is offering you a position that’s three times more on your current basic, usually requires 10 more years experience than you have with unlimited benefits and a stonking bonus, then there might be something fishy going on. This tactic is usually to get you in the interview process in the hope that further down the line when you find out what the reality of the job is and what the pay will actually be, you’re already too far gone to start again with another firm.
  • Feedback… Whether you’re currently in a position, in an interview process or even getting appraised at work, feedback is one of the most important pointers that help your performance and make you get better at whatever you do. At Oxford Knight, acting as a consultant for our candidates and giving them useful preparation and honest feedback is the way we think things should be done. As you know, that’s not always the case, particularly when some recruiters are just looking to make a quick buck. There will not be a client out there that just tells your recruiter “He/She’s a no” without any additional information. There will always be a reason for your rejection and it will greatly help you in your job hunt to know what this is. Even if you’ve been successful moving to the next round, suggestions on what you could work on and what they like to see from candidates will maximise your chances of success. If you’re stuck with a recruiter that gives you nothing, dump them. If they won’t help you get a job, don’t help them make their commission.
  • Instinct… My final comment on dealing with difficult recruiters on your job hunt is to follow your instinct. If your gut is telling you that sleazy Bob from Business XYZ is spinning you a yarn and wasting your time, you’re probably right. Your time is precious, even more so when you’re looking for a new role. Not many businesses value the integrity of their consultant’s behaviour as much as Oxford Knight do…just make sure that if they don’t, you don’t bother with them either.

Zahra Al-Nugaidi 

 Direct:        +44 (0) 20 34755012

Zahra@oxfordknight.co.uk

 

Oxford Knight is a technical recruitment agency. None of our consultants have written a line of code… yet. We apologise if this article doesn’t keep some purist happy, but we’re trying to build a new generation of technical recruitment agencies…. We listen, participate, and deliver.

Don’t Be Evil – Is Google as Innocent as it claims?

10 May 2013

By Ben Rogan| Oxford Knight

google___don__t_be_evil__wallpaper_by_dakirby309-d4idust-1024x767

 

I am often told that that people don’t want to work in finance because it is ‘Evil’ that banks have capitalised on a broken system and that hedge funds beat old ladies with clubs before stealing their pension. Well. This is the impression some developers seem to have.

In their minds, they want to work in the veritable ethical utopia that is Google, and its’ cornucopia of ethical premises seem to attract developers by the flock. In my mind at least, this seems a little hypocritical, at least on the ethical front…

Google is the most used search engine, and successful technology company in the world. Even outstripping Microsoft. It turned a noun into a verb and has one of the best international brands, technology and product out there. Not to mention it is probably the most effective way to find information in existence today.  Google consistently commits to profit neutral and even loss making services across its entire spectrum of products. Not to mention providing an entirely free, email, mapping and internet search service. Pretty amazing huh?

Google owes its popularity and success to two things, to deftly written algorithms that quickly find the information that is plugged in by a user anywhere in the world, providing a service that is second to none. And a brand, characterised by one phrase, “Don’t be Evil”. The informal motto of Google.

Don’t be evil, centres around one premise; you can make money without being evil. And this central pillar was meant to remain at the heart of the company. Paul Buchheit, the creator of Gmail said that [it was] “also a bit of a jab at a lot of the other companies, especially our competitors, who at the time, in our opinion, were kind of exploiting the users to some extent.” The evil Microsofts and Banks.

So, Google was taking the high road, “Don’t be Evil” later made it into the Google manifesto and it became part of the brand. Google wanted to lead the way, putting the users first, and making a profit from tailored advertising. In an industry full of famously cutthroat tech giants, Google was supposed to be different.

“The problem is, there is no opt out. There are too many shades of grey in a company that takes pride in its brightly coloured logo

Google’s privacy policy has changed, and this has a big impact on where your personal information ends up. Your search results, map directions, even this videos of cats you watch, it is all collected and stored. If you use android (Latitude or Ingress), even your real time location is collected and transmitted.

Google freely collects your log-on information, details of how you used the service, such as your search queries, telephone details, date and time of calls, even SMS routing information and your IP address. With the addition of G+, Google know where you are (in real time if you have android), what you like (or don’t) and probably your measurements if you have ever done some online shopping… The idea behind all this information gathering and assimilation is that Google can provide better, more “tailored services” (they mean adverts)  to their users.

The problem is, are they going too far? Right now George Orwell is turning in his grave, ‘they’ know where we are, where we frequent most, what we like and the messages we send each other. But let’s be fair this is information we chose to freely give away, for services we don’t have to use, on a completely free product. We, as users seem to have a sense of entitlement to these services that is perhaps unwarranted. Although let’s face it, is there really an alternative to Google? It might not technically have a monopoly on search, but is there really an alternative ?

I can’t pretend to be comfortable with the amount Google know about me, of the amount of my -not so personal – information they have access to, but I can’t, realistically imagine my life completely unplugged either. The problem is, there is no opt out. There are too many shades of grey in a company that takes pride in pride in its brightly coloured logo.

Eric Sdmidt the Executive Chairman of Google argued, “If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place”. This concerned me the most, it reeked of the watcher and the watched, of a bold totalitarian leviathan, it is the kind of phrase dictators placate the masses with. To say only miscreants worry about net privacy, well, that’s stupid.

We have seen the Arab Spring launch off the back of social media, China begin to open international doors, and on occasion even a glimmer of light out of North Korea. To say only miscreants should be concerned is fallacious and short sighted. Fair enough, Google can have access to the sites I visit, to construct some pretty cool ads, but what I email people? My passwords? My genetic make up? My plans for a revolution? I think not.

The boundaries between civil liberties and the appropriation of my personal data appear to be becoming rather blurred, over the last few years, Google has collected my data – a lot of it, but what else are they going to do with it ? Can they be trusted ? Google has abused users’ privacy by knowingly geo-targeting them with ads for illegal/counterfeit drugs and by not changing its privacy policy despite being told to do so by six European data protection agencies, and now faces additional legal challenges after making no change to the policy and Google+ forces consolidation of privacy policies with no opt out, not to mention recent charges of illegal web scraping.

“Don’t be evil” remains there, in the Google manifesto, in 6th position. I’m not so sure it should be there at all. For a company that brands itself as different to its competitors, with bright branding and fluffy edges, it has all the comforts of room 101.

So. Where does this leave us ? if Google, the brightest of stars is dimming, where would I recommend.

More and more these days, I find an ethical appeal in start-ups. Sure, it isn’t the easiest working environment, some lack stability, some will fail, others will go on to outgrow Google. But since when was the ethical choice the easy one? I find those two paths seldom cross.

In a start-up the company is as ethical as you make it, when you join at the embryonic stage you have the control to help it evolve into a creature you can be proud of. I work with start-ups, something I am quite proud of, companies that track diseases, locate laundered funds, some are doing groundbreaking research into DNA mapping, others building reusable gaming infrastructures.

If you have thoughts, think there is an excellent ethical player out there for developers, or think I’m downright wrong, drop me an email, or pick my brains. I’d love to hear your thoughts.

Benjamin

Ben.rogan@oxfordknight.co.uk

uk.linkedin.com/pub/benjamin-j-rogan/47/b58/527

Call on +44 (0) 20 31372967 or email me on ben.rogan@oxfordknight.co.uk

Oxford Knight is a technical recruitment agency. None of our consultants have written a line of code… yet. We apologise if this article doesn’t keep some purist happy, but we’re trying to build a new generation of technical recruitment agencies…. We listen, participate, and deliver.

 

The rise and rise of Big-Data

18 January 2013

By Sean Fanning | Oxford Knight

big-data-1024x588

Think for a moment back to when you were 18 years old. How much data did you put on the web? A few paragraphs on Myspace, the usual emails, maybe an E-Bay ad or even a blog like this. Now think of today’s 18 year old – by and large using Twitter, Facebook, Youtube etc on a daily basis, and in doing so putting huge amounts of content onto servers globally. This exponential rise in content has given rise to the phenomenon of Big-Data. Put simply, according to IBM, we now create 2.5 quintillion bytes of data/day.

What is Big-Data?
Global companies, institutions and each of us produce ever increasing amounts of data from online purchases to personal videos to climate sensors. Not only does this data need to be stored somewhere, but more importantly this data holds true value as enormous data sets in which patterns emerge and valuable, intelligent predictions can be made.

In the truly globalised digital age we inhabit data takes many forms; structured and unstructured, temporal, geospatial, disparate and unconnected. Big Data technologies not only store this data but critically allow companies to wrangle data quickly and efficiently in ways that were traditionally out of reach. Quantitative analysis has become much more accessible.

Big Data Technologies
Far and away the most famous and most-used Big Data technology is Apache Hadoop which is an open-source framework that supports data-intensive distributed applications. Hadoop is run over a large cluster of computation-independent computers and allows applications to work with petabytes of data.

Essentially the framework implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node or in the cluster. It is derived from Google’s MapReduce and contains Hadoop Kernel / MapReduce and HDFS as well as Hive, HBase, Zookeeper and so on.

Yahoo! inc were early pioneers but unsurprisingly Facebook now claim to have the largest Hadoop cluster in the world with 100PB of storage.

Practical Applications

Content Delivery – Firms that are now commonplace such as Last.fm and Netflix have a lot to thank Big Data pioneers for as it has been the key technology underpinning their growth and total accessibility. To be able to deliver huge amounts of content in real-time to distributed users at their timing request is an engineering challenge that could only be overcome by Big Data technologies. Netflix stores over 1 PB of data on Amazon’s hosting cluster in North America.

Analytics – The rise of opensource Big-Data technologies and the reducing cost of running complex computations has sprouted a rise in firms that specialise in analysing both industries, institutions and also individuals. So whether it’s Klout who analyse the impact of individuals through Social Media and monetise it; or DataSift who extract insight from billions of public data points on Twitter, there is a company analysing your public data somewhere!

Internal Analytics – The biggest companies can streamline and massively improve efficiency within their sales and organisational structure by employing quantitative analysis on a huge scale. Examples include power companies predicting power usage through analysing over 400 million power meters.


Big Data Developers

There has been a steady increase in the amount of developers who are specialising in Big-Data technologies and there is very little to suggest that this won’t increase. As the paradigms of distributed computing and Big Data take much deeper root in Academia engineers will be exposed at an earlier stage and in general the standard of engineering will improve. Moreover specific to the UK is the Valley shift to London whereby Silicon Valley innovation is mirrored and improved upon by the next generation of entrepreneurs. Indeed many of the Silicon Roundabout firms harness Big Data or specialise wholly in it meaning there are many jobs and good prospects in London for those with particular interest in data wrangling.

Future

So then, is Big Data a modern version of the DotCom bubble? While the proliferation of firms pertaining to be doing something original with data reduces and the market congests, the fundamental driving force behind it shows no signs of stopping. Data is and will continue to be essential in this modern digital age and with millions more accessing the web and creating data each day, every indication would suggest those developers that specialise in Big Data have a bright future.

Here too the market will become congested and contractors will become prevalent as the big banks and financial firms finally cotton on. Given the way technology in general is developing, there is nothing to say that we shouldn’t expect new computational paradigms to develop and flourish over time. But for now – Big Data is Big Business and it most likely will be going forward too.

Want to Find out More?

So whether you are a Big-Data specialist or an accomplished engineer looking to stay ahead of the curve and enter this exciting and challenging market, I am sure that Oxford Knight will be able to give you excellent advice and guidance. If you are a potential candidate or a new firm with a need to find relevant and specialised candidate don’t hesitate to get in touch.

By Sean Fanning

Linkedin: uk.linkedin.com/in/seanfanning1

Email: sean.fanning@oxfordknight.co.uk

Website: www.oxfordknight.co.uk

Oxford Knight is a technical recruitment agency. None of our consultants have written a line of code… yet. We apologise if this article doesn’t keep some purist happy, but we’re trying to build a new generation of technical recruitment agencies…. We listen, participate, and deliver.