My talk at Clojure Conj 2015
Tue, 17 Nov 2015 Permalink
I presented my "Clojure for Business Teams - Decomplecting Data Analysis" talk at Clojure Conj 2015 on Monday. I had some technical issues with my laptop in the beginning, but managed to get past them. The audience was incredibly patient and supportive. I really appreciate that.
The video of the talk has already been published on YouTube, and I've included a PDF of the slide deck that I used below (1).
Thanks to the awesome folks who organize Clojure Conj for inviting me to present. Do check out all the other great talks that they've posted also.
Clojure and Decomplecting data analysis for business users
Tue, 15 Sep 2015 Permalink
TL;DR Data analysis has a lot of incidental complexity. Composable abstractions can help reduce this. Clojure solves this for developers, but can this be extended to business users who are non-coders? Developers want to continue writing programs in code, but business teams need ways to discover, configure and run them directly and independently. Could a Yahoo Pipes-like approach to data analysis bridge this gap?
Analyzing data with software tools is rife with incidental complexity (1) - by which I mean the myriad data formats and shapes, algorithms, APIs and programming languages that one has to wrangle with.
Clojure's functional abstractions and composability help dramatically reduce many of these incidental complexities - at least for developers. We are able to quickly build robust and reusable pipelines of data processing code and put them together in different combinations to tackle each, seemingly one-off, problem.
That's great for us. But, in most companies, Data Analysis cuts across many functional boundaries. We need to extend the flexibility in abstraction and composition beyond developers to business teams.
The average off-the-shelf data analytics tool for business users is either severely feature constrained in the name of ease-of-use, or profusely leaks its thinly wrapped programming language abstractions in the name of power. A savvy business user who needs something more expressive than Excel, without having to become a software engineer, doesn't have very many options. (2)
I care deeply about this problem, because I have personally experienced this pain point. As a dev lead supporting a business team I've seen simple report requests turn into multi man-week software projects. The business users were frustrated waiting for weeks to see their minor report requests. And the devs were unhappy having to work on pointless reports all the time.
Essentially, developers want to continue creating their core value as software abstractions in code, and business teams need to be able to discover, configure and run them directly and independently. The UNIX shell is a classic example of this from a bygone era when business users actually wrote shell scripts. But, we can do better than the stream of bytes abstraction of the UNIX pipes, and fix some of other shell problems along the way. (3)
At juxt.io, my co-founder Panch and I are working on this exact problem, which had dogged us in our past jobs. Our approach is to give business users a powerful and extensible platform, into which developers can directly contribute their code abstractions as content. Using a visual and interactive UX business users can drag and drop functional components onto a design canvas and wire them together and compose higher order functionality.
Figure 1: juxt.io Interactive Data Analysis Workbench
Juxt.io builds upon ideas from previous systems like Yahoo Pipes, Apple Quartz Composer and MIT Scratch to create an interactive data analysis workbench in which Analytics, ML and Web API components can be composed together, using Clojure as the extension language and Clojure's rich data structures.
That's the really high level picture. I'll dive into more of the details about the implementation stack, and some of the challenges and learning in upcoming posts.
Thanks to Panch Chandrasekaran and Sujatha Jagannathan for their feedback on early drafts of this post.
1. Incidental (or accidental) complexity as opposed to the essential complexity of any given problem. See Fred Brooks' No Silver Bullet (wikipedia).
2. Yes, Excel has a Turing complete macro language but it's a huge leap from the worksheet's visual UX. There's also Excel-REPL, which is a nice idea but doesn't address the Excel side of the problem.
3. See UNIX Shell and The UNIX-Haters Handbook Chapter 8
clojure-mode and slime
Sun, 03 Mar 2013 Permalink
As a long time user of SLIME I was a bit disappointed to see clojure-mode 2.0 drop support for it in favor of nrepl. I looked into nrepl but found it to be not as feature complete as SLIME, at present. Also, I still work on some sizeable Common Lisp code, which relies entirely on SLIME, and I want to be able leverage any tooling work I do across all my projects – so SLIME wins.
As it turns out, it wasn't at all difficult to resurrect the SLIME integration code from clojure-mode 1.x and load it alongside the newer clojure-mode.
I've committed the clojure-mode-slime.el Emacs Lisp code into the following repo, along with some other Clojure/Emacs hacks:
http://github.com/kriyative/clojure-emacs-hacks
I hope this is useful to other SLIME die-hards in the Clojure community as well. Feedback and bug reports are most welcome.
Older posts
- Sat, 02 Mar 2013
- clojure, emacs, and docs redux
- Wed, 20 Feb 2013
- New Site
- Sat, 28 Jul 2012
- Emacs for Clojure - Part 2
- Sat, 21 Jul 2012
- Emacs for Clojure - Part 1
- Sat, 26 Mar 2011
- ECL for iOS update for ECL-11.1.1, Xocde 4, and SDK 4.3
- Mon, 10 Jan 2011
- clojurejs -- a Clojure (subset) to Javascript translator
- Mon, 20 Dec 2010
- Lisp functions as UI callbacks in ECL/iOS
- Tue, 14 Dec 2010
- ECL for iOS updated to work with SDK 4.2
- Tue, 23 Mar 2010
- Displaying a daily Org-mode agenda reminder in Emacs
- Wed, 10 Feb 2010
- Snapshot of ECL for iPhone
- Fri, 29 Jan 2010
- my other lisp blog
- Fri, 29 Jan 2010
- ECL on iPhone update
- Tue, 29 Dec 2009
- A lightweight `bind' macro
- Mon, 14 Dec 2009
- community funded clojure
- Sat, 12 Dec 2009
- ECL on iPhone (redux)
- Sun, 21 Jun 2009
- clojure, emacs, and javadocs
- Thu, 11 Jun 2009
- cynojure.aws -- amazon web services from clojure
- Mon, 08 Jun 2009
- cynojure - clojure library on github
- Fri, 08 May 2009
- sql and s-expressions
- Wed, 06 May 2009
- welcome to cynojure
- Wed, 06 May 2009
- keyword args in clojure