SummerOfCodeIdeas
These are a list of ideas compiled by mlpack developers; they range from simpler code maintenance tasks to difficult machine learning algorithm implementation, which means that there are suitable ideas for a wide range of student abilities and interests. The "necessary knowledge" sections can often be replaced with "willing to learn" for the easier projects, and for some of the more difficult problems, a full understanding of the description statement and some coding knowledge is sufficient.
In addition, these are not all of the possible projects which could be undertaken; new, interesting ideas are also suitable.
At the bottom are a list of projects from 2013 and 2014 that aren't really applicable to this year, or were done.
For more information on any of these projects, visit #mlpack in freenode (IRC) or email mlpack@cc.gatech.edu (see also http://lists.cc.gatech.edu/mailman/listinfo/mlpack ). It is probably worth consulting the mailing list archives first; many of these projects have been discussed extensively in previous years. Also, it is probably not a great idea to contact mentors directly (this is why their emails are not given here); a public discussion on the mailing list is preferred, if you need help or more information.
If you are looking for how to get started with mlpack, see the get involved page and the tutorials and try to compile and run simple mlpack programs. Then, you could look at the list of issues on Github and maybe you can find an easy and interesting bug to solve.
For an application template, see the Application Guide.
- Implement tree types
- Neuroevolution algorithms
- More diverse build slaves (OS X and Windows)
- Profiling for further optimization
- Automatic bindings
- Fast k-centers Algorithm & Implementation
- Improvement of tree traversers
- LMNN with LRSDP implementation
- Fixes to MVU and low-rank semidefinite programs
- Parallel stochastic optimization methods
- Essential Deep Learning Modules
- We need to go deeper - GoogLeNet
- Alternatives to neighborhood-based collaborative filtering
- Low rank/sparse optimization using Frank-Wolfe
- Approximate nearest neighbor search
- Dataset and experimentation tools
- Decision trees
description: mlpack focuses heavily on dual-tree algorithms, which each rely on a type of space tree (data structure). These tree types include kd-trees, ball trees, cover trees, vantage point trees, R trees, R* trees, metric trees, principal axis trees, k-means trees, random projection trees, Bregman ball trees, UB trees, R+ trees, Hilbert trees, X trees, segment trees, interval trees, range trees, and others. However, mlpack only implements kd-trees, cover trees, ball trees, R trees, and R* trees. This project involves implementing other tree types for mlpack. A good project will select a handful of tree types and implement them (with tests and documentation) over the course of the summer. The trees must be implemented according to this example interface so that they work with any type of dual-tree algorithm. In addition, this project could possibly contain a research component -- benchmarking runtimes of dual-tree algorithms when different types of trees are used.
difficulty: 5/10
deliverable: Implemented tree types and proof (via tests) that the tree types work with mlpack's dual-tree algorithms
necessary knowledge: Data structures, C++, knowledge of memory management
relevant tickets: #184, #275, #272, #227, #228, #289
potential mentor(s): Ryan Curtin
description: The Nintendo Entertainment System (NES) wasn't the first unforgettable gaming console most of us ever played, but with the unforgettable library of games it certainly made an iconic part of many of our childhoods. Countless hours spent in heated Mario Bros. matches, or battles against Dr. Wily's destructive robots in the Mega Man series. And who could forget their first foray into the magic world of The Legend of Zelda? This project revitalizes this passion and combines it with recent developments in evolutionary algorithms to train artificial neural networks to play some of the unforgettable games. In more detail, this project involves implementing different neuro-evolution algorithms applied to NES games for mlpack. A good project will select one or two algorithms and implement them (with tests and documentation) over the course of the summer, e.g.:
- CNE: simple weight evolution over a topologically static neural network.
- CMA-ES: Covariance Matrix Adaptation Evolutionary Strategy.
- NEAT: evolution of weights and topology from simple initial structures.
- HyperNEAT: indirect encoding of network weights, using Compositional Pattern Producing Networks.
The neuro-evolution algorithms must be implemented according to the mlpack's neural network interface so that they work with various layer and network structures. In addition, this project could possibly contain a research component -- benchmarking runtimes of different algorithms with other existing implementations.
Note: We have set up an NES emulator (FCEUX) system. We already also have written some code to remotely communicate with the NES emulator, that allows a user to initiate a connection from their machine to our machine, and open up a port back to itself over this connection to e.g. get the current frame as a matrix, get the position, set input, etc. We can also obtain a system that can be used to train the neural networks. You can find the code used to communicate with the NES emulator here: https://github.com/zoq/nes
difficulty: 7/10
deliverable: Implemented neuro-evolution algorithms and proof (via tests) that the algortihms work with mlpack's neural network structure.
necessary knowledge: a working knowledge of what neural networks are, willingness to dive into some literature on the topic, basic C++
recommendations for preparing an application: To be able to work on this you should be familiar with the source code of mlpack. We suggest that everyone who wishes to apply for this idea, try to compile the source code and explore the source code, especially the neural network code and the code to communicate with the emulator. Students should at least go through the source code of the existing neural network code to get a good idea of how much work each algorithm would involve. If you have more time, try to review the documents linked below, and in your application provide comments/questions/ideas/tradeoffs/considerations based on your brainstorming.
relevant tickets: #412, #413, #414, #555
references: Deep learning reading list, Deep learning bibliography, HyperNEAT-GGP, Evolving Neural Networks through Augmenting Topologies
potential mentor(s): Marcus Edel
description: mlpack has an automatic build server (Jenkins) running at http://knife.lugatgt.org:8088/ and also has two Windows 7 build slaves (x32 and x64). However, Jenkins is not configured properly to build mlpack on Windows. Therefore, the correct Jenkins build rules for Windows systems need to be determined and put in place, so that each revision of mlpack can be built and tested on Windows. In addition, a build server for OS X needs to be set up. The hardware can be obtained but the installation of necessary packages and setup will need to be done. Alternatively, it might be possible to use something like AppVeyor to test the Windows build without needing to maintain hardware on Jenkins. A good project proposal will address the proposed setup and its advantages and disadvantages.
difficulty: 5/10
deliverable: Windows and OS X build slaves integrated into Jenkins builds
necessary knowledge: Windows familiarity, Visual Studio, some C++ (for debugging compilation errors)
relevant tickets: none are open at this time
potential mentor(s): Ryan Curtin
description: mlpack could run even faster if it used profiling information during compilation. This entails adding extra build steps to the build process: first, to run a subset of mlpack programs with profiling information, and second, to rebuild mlpack using that profiling information. The difficulty here is choosing datasets which reflect general characteristics of the datasets which users will run mlpack methods with. If an unusual dataset is chosen, then mlpack will be compiled to run very quickly on that type of dataset -- but it will not run as well on other datasets. Another issue here is that the profiling process should not take extremely long (that is, longer than an hour or so), so we cannot choose a huge variety of large datasets.
deliverable: a 'make profile' build option which performs the profiling as described above
difficulty: 6/10
necessary knowledge: some machine learning familiarity, some CMake scripting
relevant tickets: #48
potential mentor(s): Ryan Curtin
description: A better alternative to writing bindings by hand would be to have some script which processed machine learning method information and automatically generated bindings for MATLAB, Python, R, and perhaps other languages. SWIG, unfortunately, is not really an option for this. This project is somewhat exploratory and may not end up producing results if nothing is available, so this project will be research-intensive. In all likelihood the project will involve building a binding generator which can take in mlpack executable code (such as src/mlpack/methods/neighbor_search/allknn_main.cpp) and can produce a binding to another language that gives the same options as the command-line interface.
deliverable: automatic binding generators for mlpack methods which are easy to maintain
difficulty: 6/10
necessary knowledge: a working knowledge of what bindings are and a good ability to do research on software
relevant tickets: #202
description: The k-centers problem is a fundamental part of many machine learning algorithms (See Cortes & Scott, "Scalable Sparse Approximation of a Sample Mean", 2013 for a recent use of it). The basic problem is: given a set of points and integer k, choose k points that minimize the maximum distance between a point in the set and its nearest neighbor in the set. The ideas underlying the Dual-Tree Boruvka MST algorithm in mlpack could probably be extended to provide an efficient implementation for this problem. This one is more research / algorithm oriented, but would definitely require some implementation to evaluate the idea.
deliverable: working, tested k-centers implementation; paper comparing it to basic Gonzales algorithm and other implementations
difficulty: 7/10, but biased by being really related to my (Bill's) thesis work
necessary knowledge: some understanding of geometry and spatial data structures, willingness to dive into some literature on the topic, basic C++
relevant tickets: none open at this time
potential mentor(s): Bill March
description: Many of mlpack's machine learning methods are dual-tree algorithms. These dual-tree algorithms are abstracted in such a way that they all use the same tree traversal methods. If these traversers could be improved, this would result in runtime gains for all tree-based mlpack methods. This requires a lot of abstract thought in very weird ways and often debugging and profiling tree traversers is fraught with sadness.
deliverable: demonstrably improved runtime for tree-based methods
difficulty: 9/10
necessary knowledge: very in-depth C++ knowledge and understanding of tree structures; familiarity with machine learning methods is helpful
relevant tickets: #235
potential mentor(s): Ryan Curtin
description: mlpack has a working LRSDP (low-rank semidefinite program) implementation, which gives faster solutions to SDPs. This could give good speedups for existing SDP-based machine learning methods, such as LMNN (large margin nearest neighbor). This project should only entail the expression of LMNN as a low-rank SDP and then the implementation should be straightforward because the mlpack LRSDP API is straightforward. The difficulty is still quite high, though, because debugging LRSDPs is complex.
deliverable: working, tested LMNN implementation with extensive documentation
difficulty: 9/10
necessary knowledge: in-depth C++ knowledge, understanding of semidefinite programs; familiarity with LMNN is helpful
relevant tickets: none open at this time
potential mentor(s): Ryan Curtin
description: This project is not for the faint of heart. For some time now, MVU (maximum variance unfolding), a dimensionality reduction technique, has not been converging even on simple datasets. mlpack's implementation of MVU uses LRSDP (low-rank semidefinite programs); there is not another existing implementation of MVU+LRSDP. Many tears will be shed trying to debug this. A good approach will probably be to compare mlpack MVU results on exceedingly simple problems with other MVU implementation results. The final outcome of this project may not even be a successful converging algorithm but instead more information on what is going wrong.
deliverable: working MVU implementation, or, further details and information on the problem
difficulty: 10/10
necessary knowledge: understanding of convexity and duality, knowledge of semidefinite programs, incredible determination and perseverance
potential mentor(s): Ryan Curtin
description: Recently, with the prevalence of multicore machines, there has been a large interest in parallelizing stochastic gradient methods such as SGD and SCD. While in principle these parallel implementations are quite simple, in practice there are lots of subtle details that need to be handled. Examples include, how do I anneal my step size, how much coordination/synchronization should I use, how should I partition my data in memory for good cache locality, do I solve the primal/dual, etc. This project would implement parallel SGD/SCD, and explore some of these design decisions.
deliverable: parallel implementations of stochastic gradient descent and stochastic coordinate descent, including both test cases and possibly an empirical analysis of the convergence behavior on various datasets for the issues discussed above.
difficulty: 5/10
necessary knowledge: C++, experience with writing multithreaded programs, some background in convex optimization would also be helpful
potential mentor(s): Ryan Curtin
description: In the past years Deep Learning has markedly attracted a lot of attention in the Machine Learning community for its ability to learn features that allow high performance in a variety of tasks. For example, DeepMind has shown that neural networks can learn to play Atari games just by observing large amounts of images, without being trained explicitly on how to play games. This project involves implementing essential building blocks of deep learning algorithms based on the existing neural network codebase. A good project will select a handful of architectures and implement them (with tests and documentation) over the course of the summer. This could include Restricted Boltzmann Machines (RBM) and training algorithms, Deep Belief Networks (DBN), Radial Basis Function Networks (RBFN) and Bidrectional Recurrent networks (BRN). The architecture should be created to build a foundation for integrating many more models including support for state-of-the-art deep learning techniques.
deliverable: Implemented deep learning modules and proof (via tests) that the code works.
difficulty: 5/10
necessary knowledge: a working knowledge of what neural networks are, willingness to dive into some literature on the topic, basic C++
recommendations for preparing an application: Being familiar with the mlpack codebase, especially with existing neural network code is the first step if you wish to take up this task. We suggest that you build mlpack on your system and explore the functionalities. Take a look at the different layers and basic network structures. When you prepare your application, provide some comments/ideas/tradeoffs/considerations about your decision process, when choosing the models you want to implement over the summer.
relevant tickets: #412, #413, #414, #555
references: Deep learning reading list, Deep learning bibliography
potential mentor(s): Marcus Edel
description: Convolutional neural networks are the state-of-the-art algorithms for image classification. So it's not surprisingly, that almost everyone used some form of convolutional net for the annually held ImageNet competition. In 2012, it was won by DNNResearch using the convolutional neural network approach described in the paper by Krizhevsky et al. Last year the team named GoogLeNet placed first in the classification and detection challenges. The GoogLeNet team used a deep learning architecture that is more convolutional than previous network architectures which can increase the depth and width of the network without increasing significant the parameters, using a new module called inception. This project would involve implementing the components of the GoogLeNet architecture and afterwards evaluating the network on a smaller subset of the ImageNet data. While in principle, the basic implementation of the main module for the GoogLeNet the inception layer is quite simple, there are many fun spinoffs which has to be covered such as:
- Addressing the problem of generating possible object locations for use in object recognition step. The GoogLeNet Team used the Selective Search algorithm, which combines the strength of both an exhaustive search and segmentation in combination with the multi-box approach.
- Addressing the initialization of the network parameters. Since GoogLeNet contains over 100 individual models, the model initialization is tricky. Bad Initializations would produce no gradient for the learning process.
- Addressing the special structure which isn't linear at all, there are auxiliary classifiers connected to the intermediate layers that get also back propagated, to and provide additional regularization.
This project involves some really neat methods, and it would be great if mlpack provided a solid implementation of this architecture including a method to generate possible object locations.
deliverable: Implemented GoogLeNet architecture including a method to generate possible object locations such as selective search and proof (via tests) that the algorithms work with mlpack's neural network structure.
difficulty: 8/10
necessary knowledge: Some background in deep learning would be very useful. While the actual algorithms are simple, to really understand what is going on requires some level of familiarity with the math. But detailed knowledge of the methods is not necessary - only a willingness to read some documentation and papers.
recommendations for preparing an application: Being familiar with the mlpack codebase is important if you like to take up this idea. We suggest that you build the library and explore the functionality that needs to be touched during the summer. Another step would be to review the documents below, and to provide comments/questions/ideas/tradeoffs/considerations in your application and outline the primary steps you would follow.
relevant tickets: #412, #413, #414, #555
references:
- Going Deeper with Convolutions Description of the deep convolutional neural network architecture named "Inception".
- Selective Search for Object Recognition Description of the selective search algorithm that addresses the problem of generating possible object locations.
- Deep learning reading list, Deep learning bibliography
potential mentor(s): Marcus Edel
description: The past two years in Summer of Code have seen the addition of a highly flexible collaborative filtering framework to mlpack. This framework offers numerous different types of matrix factorizations, and is likely the most flexible package for this (with respect to matrix factorizations). However, the implementation uses k-nearest-neighbors to select its recommendations, whereas there are more options. This project entails the investigation of alternatives (for example, weighted nearest neighbors and regression techniques) and the implementation of these alternatives in a flexible manner.
deliverable: Implemented alternatives to k-NN for recommendation selection
difficulty: 7/10
necessary knowledge: experience with C++ and templates, knowledge of recommendation systems, willingness to read the literature
relevant tickets: #406
references: this paper describes an alternative to k-NN recommendation selection: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.129.4662&rep=rep1&type=pdf
potential mentor(s): Ryan Curtin, Sumedh Ghaisas
description: The past decade has seen an explosion of work focusing on underdetermined linear inverse problems, where the solution is assume to have some sparse structure. Compressed sensing, matrix recovery, lasso, etc all fall under this framework. Many specialized algorithms have been developed for particular problem instances. Recently, researchers have realized that many of these algorithms are actually instances of the same Frank-Wolfe algorithm. The Frank-Wolfe algorithm is an old algorithm to optimize a general convex function over a compact set. Martin Jaggi in this awesome paper showed how algorithms such as orthogonal matching pursuit and various low rank matrix recovery algorithms are simply instances of Frank-Wolfe, and proved general O(1/k) convergence rates. This is really neat stuff, and it would be great if mlpack provided a solid implementation of this framework.
deliverable: An implementation of Frank-Wolfe for the various atomic norms listed in Jaggi's paper, complete with test cases. The implementation should be general enough that we can recovery OMP by simply setting a few template parameters.
difficulty: 7/10
necessary knowledge: Some background in convex optimization would be very useful. While the actual algorithms are very simple, to really understand what is going on (which helps in debugging, for instance) requires some level of familiarity with the math.
relevant tickets: None at this time
potential mentor(s): Ryan Curtin
description: mlpack provides an extensible, flexible exact nearest neighbor search implementation in the src/mlpack/methods/neighbor_search/ directory. This is a state-of-the-art implementation, with the ability to do dual-tree nearest neighbor search instead of single-tree nearest neighbor search, like other packages (i.e. FLANN, ANN, scikit-learn) do. However, there is currently no support for approximate nearest neighbor search, despite the popularity of approximate nearest neighbor search. This is a relatively simple extension to the existing NeighborSearch code. This project would have two components: the first component would be the implementation of approximate nearest neighbor search and either modification of the existing mlpack_allknn program or implementation of a new approximate nearest neighbor search command-line program. The second component would be working with the benchmarking system (https://github.com/zoq/benchmarks/) in order to produce rigorous comparisons between mlpack's approximate nearest neighbor search implementation and other libraries, such as FLANN, ANN, or LSHKIT (and even mlpack's LSH implementation). Depending on the robustness of the comparisons, this could result in a nice publication exploring the empirical performance of approximate nearest neighbor search schemes as implemented by existing machine learning libraries.
The figure below is some output from the existing benchmarking system on exact nearest neighbor search (from http://mlpack.org/benchmarks.html ); ideally, this project should result in some similar nice figures.
deliverable: working approximate kNN code with tests, benchmarking scripts, benchmark comparison numbers and figures
difficulty: 6/10
necessary knowledge: C++, Python, background on nearest neighbor search techniques
relevant tickets: (have not filled this out yet)
potential mentor(s): Ryan Curtin, Bill March
description: I would say that 90% of any machine learning problem is getting the data into a workable format. Real-world data is noisy, has missing values, is encoded improperly, is in weird formats, and has all kinds of other problems. Diagnosing and fixing these issues, and then preparing the data for input into a machine learning system, is often the most time-consuming part of the process. Usually, this is done with a conglomeration of Python tools (like pandas for instance), but it would be useful if mlpack users did not need to preprocess their data in other software tools before putting it into mlpack. Thus, there are some useful pieces of functionality that could be added to mlpack:
- checking a dataset for loading problems and printing errors
- imputation strategies for missing variables
- splitting a dataset into a training and test set
- converting categorical features into binary features (or numeric features)
There should be a C++ API for these tools, but also a command-line program that could be used. Below is an example gif of what the interface might be able to do; this can be used as inspiration for a project proposal.
deliverable: command-line program for dataset tools and C++ API, plus extensive documentation and tutorials on how to use it
difficulty: 3/10
necessary knowledge: C++, preferably familiarity with data science workflows and tools
relevant tickets: none at this time
potential mentor(s): Ryan Curtin
recommendations for preparing an application: This is an open-ended project and a student should identify in their proposal which problems they intend to solve and clearly illustrate how they intend to solve them. Should this project be completed, it will likely see a high amount of use, so clear and comprehensive documentation is necessary. That means that the project proposal should be equally clear on how each of these goals should be accomplished. There is room for flexibility here, so, if the vision outlined above does not mesh with a vision that you have for this project, that's okay; let's discuss ideas!
description: During Google Summer of Code 2014, Udit Saxena implemented decision stumps as part of his AdaBoost project. More recently, an implementation of streaming decision trees (Hoeffding trees) has been added. What would complement these things well would be to extend the implementation of decision stumps to full-blown decision trees (like ID3, CART, or C4.5). This could be done in one of several ways: adapting the existing decision stump code, writing a new class, or adapting the tree construction algorithm in density estimation trees (src/mlpack/methods/det/) to handle arbitrary loss functions. Other ideas might include implementation streaming Mondrian trees (http://papers.nips.cc/paper/5234-mondrian-forests-efficient-online-random-forests.pdf), but the overall goal here is to boost the number of classification techniques that mlpack has available.
deliverables: implementation of decision trees or similar algorithms, with tests and benchmarks comparing against other libraries
difficulty: 5/10
relevant tickets: none open at this time
potential mentor(s): Parikshit Ram, Nick Vasiloglou, Ryan Curtin
recommendations for preparing an application: A good candidate should be familiar with all of the decision tree and density estimation tree functionality in mlpack, in order to make a good decision about how it could be refactored or integrated into one class (or if that is even possible). API design is one of the most important things here, so the proposal should definitely have a section focusing on the API of the finished project.
description: Currently, there are MATLAB bindings contributed by Patrick Mason, but they are not comprehensive; there are some mlpack methods which do not have MATLAB bindings. This project entails developing the rest of the mlpack MATLAB bindings and standardizing them. Detailed knowledge of the machine learning methods is not necessary for this project -- only a willingness to read some documentation. Regardless, someone working on this project can expect to, over the course of the project, become at least somewhat familiar with state-of-the-art machine learning methods and how they work. Each binding consists of a MATLAB script which provides documentation and handles input, and a MEX file which passes the input to mlpack methods.
deliverable: MATLAB bindings for all mlpack methods (about twenty) with standardized APIs
difficulty: 2/10
necessary knowledge: MATLAB familiarity and some C++
why not this year? Realistically, the automatic bindings project is far more useful and interesting, and reduces the maintenance load of bindings significantly.
description: Similar to the MATLAB bindings, Python and R bindings would be useful for mlpack users who don't know C++ or would prefer to use Python or R to do large-scale machine learning. This project entails developing bindings for Python and R which have consistent API and documentation. Detailed knowledge of the machine learning methods is not necessary -- only a willingness to read some documentation. Regardless, someone working on this project can expect to, over the course of the project, become at least somewhat familiar with state-of-the-art machine learning methods and how they work. Because no previous work has been done on these bindings, an investigation into options for bindings and the necessary CMake configuration to build the scripts.
deliverable: Python bindings or R bindings (or both) for all mlpack methods (about twenty) with standardized APIs
difficulty: 3/10
necessary knowledge: Python knowledge, R knowledge, and some C++; CMake knowledge would be helpful but is not necessary
relevant tickets: #202
why not this year? Like the MATLAB bindings, this project is superseded by automatic bindings.
description: For widespread adoption of mlpack to happen, it is very important that relevant and up-to-date benchmarks are available. This project entails writing support scripts which will run mlpack methods on a variety of datasets (from small to large) and produce runtime numbers. The benchmarking scripts will also run the same machine learning methods from other machine learning libraries (see OtherLibraries) and then produce runtime graphs. This can be integrated into Jenkins so that benchmarks are auto-generated nightly, which could be very helpful in informing developers which of their changesets have caused speedups or slowdowns.
deliverable: an automated benchmarking system built into Jenkins
difficulty: 7/10
necessary knowledge: bash scripting (or other scripting languages; Python could work), some machine learning familiarity
why not this year? This project was completed as part of GSoC 2013 by Marcus Edel and improved in 2014 by Anand Soni. It's very cool; see http://www.mlpack.org/benchmark.html .
description: mlpack is packaged in Fedora and RHEL, but currently not in Debian due to the stringent packaging requirements. To complete this project, mlpack would need to be properly packaged for Debian and Ubuntu and submitted in the proper manner. This would involve a package review process and probably would result in the project's undertaker becoming a Debian contributor or Debian developer. Because this process is generally somewhat slow and depends on other people's input, it may be best combined with another simple, low-difficulty project.
deliverable: mlpack in Debian or Ubuntu repositories (or at least an in-motion process)
difficulty: 2/10 but includes a lot of reading documentation
necessary knowledge: Linux familiarity, knowledge of how the open-source community works, and shell scripting
relevant tickets: #327
why not this year? This is far too little work for a good GSoC project. It could be part of another project, though.