Introduction
This article is about making machine learning tools more accessible.
But, it is not about dumbing them down for lay audiences or about creating toy interfaces for toy problems.
Though there are many differences between human and machine learners, every human being has ample experience in the most crucial aspect of machine learning — we know what it means to learn from experience.
In fact, we are so immersed in this process that the greatest challenge in teaching others is remembering how we came to learn something ourselves.
This metacognitive ability is a precious and hard-won skill that is especially pertinent to the work of educators but is also a key component of everyone’s day-to-day interactions with others.
When we teach our children to ride bikes, we do not rattle off exact programmatic procedures. We demonstrate, we watch as they approximate the action, and we then direct the child’s focus to overlooked aspects of the task.
Machine learning tools should help us to do exactly those things for machines so that they can in turn help us to perform tasks too complex to be expressed by conventional code.
Tools that help us to sharpen these metacognitive abilities will not only aid us in building more useful machines, they will help us to understand ourselves better and engage with the world more effectively.
So, when I say ‘accessible,’ I mean that machine learning tools should be able to serve as an extension of every human’s innate capacity to both learn and teach through experience.
Machine learning is a tool for everyone.
In its current form, though, this is certainly not obvious.
Presently, the exploration of machine learning requires programming skills, a working knowledge of machine learning techniques and the underlying math, and even some DevOps skills. In combination, these requirements present a high barrier to entry.
The first generations of machine learning tools have focused on aiding expert practitioners in managing high-performance pipelines at a highly granular level. These tools have abstracted the conceptual demands of machine learning into conventional programmatic structures, despite an inherent incongruity between the deductive mentality of conventional code and the inductive mentality of machine learning systems.
These tools do not accentuate the skill of curriculum design, or the process of curating a learner’s experience of a complex concept by distilling and bringing forward its most meaningful elements.
In my teaching work, I have found that a student’s programming and math background is often not the best predictor of how they will take to applied creative work with machine learning. In many cases, the most captivating projects have come from the clearest writers and the most careful designers.
This is because so much of what makes a machine learning project successful relates to the student’s articulation of a learning problem — how a particular concept can be demonstrated through a particular presentation of data.
As machine learning algorithms and mechanisms for automated architecture and hyperparameter search become more robust, some of the skills currently necessary to machine learning work will become less vital while the skill of clearly articulating a learning problem will become all the more.
Ideally, the next generation of machine learning tools will not only help novices to become acquainted with these problem-solving mechanisms. It will also help experts to explore machine learning concepts more effectively through tools that have been designed specifically around the mindset of machine learning.
Traditionally, a programming language is treated as distinct from the interactive development environments (IDEs) and other tools that play a role in the process of writing software.
These delineations were already outdated vestiges of the pre-graphical era in computing.
But, in the machine learning age, they will soon become critical hindrances to the work of software developers.
The need for more holistic programming environments can already be seen in the efforts of DeepMind, OpenAI, Microsoft and others to build simulation environments for training reinforcement learning systems.
Jupyter notebooks, Matlab, Mathematica, Swift, Processing, Scratch and web development tools have helped to blur these distinctions by moving programming beyond the confines of pure text editors into more cohesive environments that seek to accommodate the interactive and multimedia aspects of software development.
Yet, there is still a long way to go.
To be clear, this is not only a matter of making programming interfaces more palatable to their human users, though that is a worthy goal in itself.
The broader imperative comes from the fact that machine learning is forcing computer programs out of the darkened basement and into the visual, auditory and physical world.
Deeper mechanisms for communication should not be tacked onto existing interfaces or relegated to auxiliary libraries. They must be brought to the very center of our tools.
Going further, we should take the paradigmatic shift brought on by machine learning as an opportunity to rethink numerous aspects of the software development process.
Foil
I’d like to show you a project I’ve been working on as an example of how these principles could be brought into next generation machine learning tools.
Some of the elements mentioned below have already been implemented and others are in the roadmap. But, for the purposes of this article, I’d like to discuss the whole vision.
Foil is a combined machine learning and creative coding platform, toolkit and IDE.
It is an attempt to close the loop between programming, machine learning and design tools in a single platform that provides a seamless path from novice to expert.
Foil envisions a multi-tiered approach in which lower-level features can be used directly by experts but are also composited together to form higher-level features for less experienced users.
As users gain expertise, they can inspect the components that comprise these higher-level features and begin to use the lower-level components as well as create and share their own higher-level compositions.
Foil’s expert mode is roughly equivalent to some existing machine learning development environments such as Jupyter notebooks, but strives to provide a more seamless integration between machine learning functionality and a range of GUI, scenegraph and data visualization functionality.
From a tool-making perspective, this mode serves as a bridge between the most granular functionality offered by Foil’s own text-based programming language and the native machine learning and creative coding toolkits that Foil utilizes under the hood. It also creates the bedrock upon which the higher-level “intermediate” and “novice” mode tools are built.
Next, I’d like to discuss how this multi-tier approach is enacted through building blocks for the construction of machine learning models and for data wrangling. I’ll also introduce a concept I call Directives, a conversational approach to project development. Finally, I’ll discuss how these components relate to a set of creative coding tools.
Machine Learning Building Blocks
Perhaps the most intimidating thing about entering machine learning work is the process of gaining an intuition for the role that certain mathematical operations play in mechanizing inductive learning processes.
But, I dare say that the challenge exists not only for the mathematically uninitiated. One piece of evidence to support this claim is the three hundred year gap between Leibniz’ first mention of the Chain Rule in calculus and the advent of the Backpropagation algorithm.
That is, even with an understanding of the underlying mechanics and mathematical tools, it is not always obvious that a particular application of those tools will yield useful capabilities when applied to systems too complex to compute mentally.
To overcome this challenge, we must lower the barrier to experimentation.
Of course, unguided experimentation is unlikely to arrive from scratch at such complex formations as those that comprise modern neural architectures.
Rather, the experimenter can be most productive by starting from known mechanisms and move outward by attempting to connect those general forms with the project ideas that have motivated the experimenter to explore this field in the first place.
This process is significantly more difficult when the examples that experimenter seeks to learn from have been codified in obscure ways by expert practitioners.
For this reason, I believe it is important to take a multi-tier approach in presenting architectural components for building machine learning systems.
To aid novices, the tool should provide a set of extremely high-level building blocks that address the basic categories of learning problems.
At an early stage in his or her experimentation, it is unlikely that the user will have need for more specialized building blocks.
As such, a high-level interface that provides easy access to a series of lightly customizable templates as well as under-the-hood automations will help the user to develop an understanding of problem-solving in this domain without inhibiting his or her early explorations.
This interface should provide graphical representations of the building blocks and their component properties in order to facilitate experimentation without the need for external reference materials and enable users to inspect the underlying architecture of the systems they have built.
As users’ intuition for and knowledge of machine learning grow, their ability to conceive of more novel architectures and mechanisms for interfacing with real-world problems will too.
They should not be forced to graduate to a completely different, more professional tool. Instead, the tool should be able to gradually expose more advanced functionality and allow users to become involved in more granular decisions.
Eventually, the advanced user may find it most expedient to interact with the tool through purely textual commands. Here too, users should not be forced to graduate to a completely different development environment.
As an extension of these building blocks, the tool should provide an extensive model zoo and the infrastructure to enable users to easily share their models.
Foil envisions three kinds of models within the Model Zoo.
Specified and trained models can be pulled off the shelf and used directly to perform a specific machine learning task such as image recognition. These models should also come with hooks that simplify transfer learning processes.
Other models will provide a pre-specified architecture, but come untrained so that the user can easily couple them with their own data sets.
Finally, an unspecified and untrained group of models will provide a set of common machine learning architectures that can be customized to the user’s own needs before training on their custom dataset.
To facilitate that process, the tool should provide seamless workflows for training models locally or in the cloud without the need for extensive DevOps knowledge.
Data Manipulation Building Blocks
Given the primacy of data in constructing machine learning systems, development tools should provide a robust set of building blocks to facilitate the loading and manipulating of data.
Though array manipulation is one of the most common tasks performed by computer scientists, it tends to be one of the greatest sources of frustration for novice programmers.
Even for a seasoned developer, it can be time-consuming and tedious to make sense of ad-hoc code for slicing complex data.
Depending on what the user wants to learn from the data, or what task she wants to perform through it, the question of how to slice the data and what neural architecture to use are interrelated considerations.
For this reason, it is important for data manipulation components to be closely aligned with the tool’s architectural building blocks.
At the highest level, the tool should provide interactive mechanisms for selecting examples from a set of files or a media stream.
It should provide graphical interfaces for building blocks that visually demonstrate how incoming data will be sliced.
At a slightly lower level, the tool should provide data loaders and common slicer types for a range of media formats as well as common open-source datasets.
At the lowest level, the tool should also provide conventional mechanisms for experienced developers to access the filesystem and manipulate numeric arrays programmatically.
Much like the architectural components, the tool should provide infrastructure for sharing custom datasets, slicers and loaders with other users.
To support dataset maintenance, the tool should provide a set of customizable watchdogs that automatically check for updates to a file directory or remote repository and perform automatic retraining of a model when updates occur.
Finally, the tool should provide streamlined mechanisms within the development environment for soliciting crowdsourced data acquisition and cleaning.
Directives
In a conventional programming environment, the developer starts with a nearly blank slate and must conceive of a set of interrelated abstractions that address the various components of the desired programmatic behavior.
The open-endedness of this medium is a wonderful thing. But it requires developers to navigate on their own each component decision in relation to each other and to the whole.
Directives provide a mechanism for helping developers to structure their thinking without imposing any one specific abstraction on it.
In the most basic sense, directives are comprised of a set of widgets for soliciting multimedia inputs and presenting multimedia outputs to the user.
The author of a directive uses these widgets to compose a kind of decision tree or guided process through which its user can respond to each decision point in the most appropriate modality for the task at hand. Visual information can be communicated visually and textual information can be communicated textually.
When a specific numeric value is needed from the user, a slider may be presented so that the user can experiment within a specific range and see its effect applied immediately and in context.
Rather than offering one preordained mental model, a Directives Zoo would provide developers with a range of possible abstractions for a variety of component problems.
Alternately, the developer can write their own directives. This process of creating a multimodal decision tree pushes the user to think clearly and purposefully through a set of decision points. Once these decision points are defined, the solution is well on its way, much like how formulating a question is often half the work of answering it.
Directives can be used to facilitate software development processes as a kind of crowd-sourced design pattern. But they can also be employed within user-facing applications to facilitate interactions with trained machine learning models or other features of the application.
In this sense, directives serve to blur the distinction between the acts of writing and using software. In the context of writing software, they help to create a more manageable, conversational experience for the developer by breaking the process into smaller tasks. In a user-facing context, they provide a more natural format for the kinds of multimodal interactions we will have with intelligent applications.
Directives can be written explicitly as code or created using directive-making directives that pose questions like, “What should the user do next?” How meta!
A directive can also be configured to provide under-the-hood data collection of user responses in order to feed a learning decision tree, much like the digital version of the game 20 Questions.
This feature brings new value to the sharing of open-source software. If a developer chooses to share a directive, her software will be improved every time someone else uses it.
Creative Coding
One of machine learning’s greatest benefits is that it enables us to move beyond the confines of the keyboard and mouse and communicate with machines in ever-more natural and intuitive ways through the machine’s increasing ability to understand the contents of images, audio streams, videos and the like.
These forms of media are critical to the training of machine learning models as well as to our subsequent interactions with trained models.
Yet, contemporary programming tools provide little assistance.
The arduous task of organizing a large multimedia dataset has been left to outside tools — requiring the developer to navigate this process through the OS, conventional design tools and ad-hoc code.
Developers generally must also cobble together numerous third-party libraries in order to visualize training metrics as well as to produce user interfaces and other aspects of an application’s front-end experience.
In order to provide an integrated development and deployment workflow and facilitate the production of modern machine-learning-enhanced applications, it is essential for the tool to provide a full creative coding framework including GUI, scenegraph and data visualization components directly within the development studio.
These creative coding tools are intended to aid developers in three distinct ways.
First, by providing graphical interfaces, directives and programmatic hooks for manipulating and organizing multimedia assets, Foil serves as a flexible design tool for creating training curriculums within the same environment used to develop the machine learning models themselves.
This functionality constitutes Foil’s role in facilitating “multimedia data as input.”
Equal importance is given to Foil’s role in creating “multimedia outputs.”
The creative coding framework enables developers to create a wide-range of interactive, user-facing applications that utilize the intelligent behaviors of custom or open-source machine learning models.
For some machine learning applications, RESTful services and messaging protocols to bridge independent systems work well enough.
But for a growing list of high-bandwidth applications, this approach can produce technical bottlenecks and impede creative exploration.
By integrating a full suite of both machine learning and front-end development tools into a single environment, it becomes feasible for an individual developer to set their sights on a wide range of high-performance, machine-learning-enabled projects.
For example, a pose estimation model could be employed to provide Kinect-like skeletal control of a character within a user-created game or an image segmentation model could be employed within a user-created image editing tool.
Finally, to complete the loop, developers can use multimedia outputs to generate new input training data by establishing scenegraph listeners and other real-time watchers that capture user or entity behaviors during runtime.
For example, user action histories could be mined for common usage patterns so that the software could suggest possible next steps.
In an application that includes a 3D physics simulation, the behavior of scenegraph nodes could be used as training data for a reinforcement learning model or robotics project.
Applications built within the Foil environment should be easily deployable to desktop and mobile systems, the web or cloud-based services, and embedded IoT devices.
Conclusion
The original personal computing revolution took hold when the tools reached a level of maturity that enabled users of all kinds to begin addressing their own needs through computers.
For this reason, I think it is reasonable to conclude that the machine learning revolution hasn’t really begun.
The true promise of machine learning lies in tools that will enable the seasoned software developer, small business owner and average consumer to see ways in which machine learning can aid their specific needs and help them develop solutions themselves.
Like the personal and mobile computer markets before it, this market will be truly immense.
This market does not yet exist, however, because the tools have not yet evolved in such a way that makes their relevance self-evident.
In the not-too-distant future, machine learning will help us to solve innumerable problems too specific to our own lives to be addressed by the ready-made functionality of a pre-built service.
It will help us to build a world that we can’t yet begin to imagine.
But we can start by imagining what such tools would look like and how we can make them both expressive and approachable.
There is no one mechanism, no singular approach or interface that will be perfect for every user or every use case.
Perhaps the only universal solution is to provide flexibility. In the age of machine learning, the toolmaker’s role is a translational one. The tool must be aligned with the experiential nature of both humans and learning machines.
Provide some starting points. Foster experimentation. Help people to structure their thinking, let them explore and encounter some complicating factors. When they do, provide a range of possible solutions. But above all, provide the infrastructure for them to produce their own solutions and to share them with others.
Enable users to adapt the tool and to be adapted by it — to grow with it.
If you enjoyed this article, you might also like my earlier related piece, “Rethinking Design Tools in the Age of Machine Learning.”
This article is based on a talk I recently gave at the Google PAIR UX Symposium.
If you’d like to follow my work on machine learning, artificial intelligence, design tools and programming languages, follow me on Medium @Patrick Hebron or on Twitter @PatrickHebron. My homepage also includes several other ways of getting in touch.