Grenade! Dependently Typed Neural Networks

we explored one of the most complex topics I’ve presented on this blog. We examined potential runtime failures that can occur when using Tensor Flow. These included mismatched dimensions and missing placeholders. In an ideal world, we would catch these issues at compile time instead. At its current stage, the Haskell Tensor Flow library doesn’t support that. But we demonstrated that it is possible to add a layer to do this by using dependent types. In the last couple weeks Now, I’m still very much of a novice at dependent types, so the solutions I presented were rather clunky. This week I’ll show a better example of this concept from a different library. The library uses dependent types everywhere. It allows us to build verifiably-valid neural networks with extreme concision. So let’s dive in and see what it’s all about! Grenade Shapes and Layers The first thing to learn with this library is the two concepts of Shapes and Layers. Shapes are best compared to tensors from Tensor Flow, except that they exist at the type level. In Tensor Flow we could build tensors with arbitrary dimensions. Grenade currently only supports up to three dimensions. So the different shape types either start with , or , depending on the dimensionality of the shape. Then each of these type constructors take a set of natural number parameters. So the following are all valid “Shape” types within Grenade: D1, D2 D3 D1 5D2 4 12D3 8 10 2 The first represents a vector with 5 elements. The second represents a matrix with 4 rows and 12 columns. And the third represents an 8x10x2 matrix (or tensor, if you like). The different numbers represent those values , not the term level. If this seems confusing, here’s a that goes into more depth about the basics of dependent types. The most important idea is that something of type can have 5 elements. A vector of 4 or 6 elements will not type-check. at the type level good tutorial D1 5 only So now that we know about shapes, let’s examine layers. Layers describe relationships between our shapes. They encapsulate the transformations that happen on our data. The following are all valid layer types: ReluFullyConnected 10 20Convolution 1 10 5 5 1 1 The layer describes a layer that takes in data of any kind of shape and outputs the same shape. In between, it applies the activation function to the input data. Since it doesn’t change the shape, it doesn’t need any parameters. Relu relu A layer represents the canonical layer of a neural network. It has two parameters, one for the number of input neurons and one for the number of output neurons. In this case, the layer will take 10 inputs and produce 20 outputs. FullyConnected A layer represents a 2D convolution like we saw with our network. This particular example has 1 input feature, 10 output features, uses a 5x5 patch size, and a 1x1 patch offset. Convolution MNIST Describing a Network Now that we have a basic grasp on shapes and layers, we can see how they fit together to create a full network. A network type has two type parameters. The second parameter is a list of the shapes that our data takes at any given point throughout the network. The first parameter is a list of the layers representing the transformations on the data. So let’s say we wanted to describe a very simple network. It will take 4 inputs and produce 10 outputs using a fully connected layer. Then it will perform an activation. This network looks like this: Relu type SimpleNetwork = Network ‘[FullyConnected 4 10, Relu] ‘[ ‘D1 4, ‘D1 10, ‘D1 10] The apostrophes in front of the lists and terms indicated that these are promoted constructors. So they are types instead of terms. To “read” this type, we start with the first data format. We go to each successive data format by applying the transformation layer. So for instance we start with a 4-vector, and transform it into a 10-vector with a fully-connected layer. Then we transform that 10-vector into another 10-vector by applying . That’s all there is to it! We could apply another layer onto this that will have 3 outputs like so: D1 relu FullyConnected type SimpleNetwork = Network ‘[FullyConnected 4 10, Relu, FullyConnected 10 3] ‘[ ‘D1 4, ‘D1 10, ‘D1 10, `D1 3] Let’s look at MNIST to see a more complicated example. We’ll start with a 28x28 image of data. Then we’ll perform the convolution layer I mentioned above. This gives us a 3-dimensional tensor of size 24x24x10. Then we can perform 2x2 max pooling on this, resulting in a 12x12x10 tensor. Finally, we can apply an layer, which keeps it at the same size: Relu type MNISTStart = MNISTStart ‘[Convolution 1 10 5 5 1 1, Pooling 2 2 2 2, Relu] ‘[D2 28 28, D3 24 24 10, D3 12 12 10, D3 12 12 10] Here’s what a full MNIST example might look like (per the on the library’s Github page): README type MNIST = Network '[ Convolution 1 10 5 5 1 1, Pooling 2 2 2 2, Relu , Convolution 10 16 5 5 1 1, Pooling 2 2 2 2, FlattenLayer, Relu , FullyConnected 256 80, Logit, FullyConnected 80 10, Logit] '[ 'D2 28 28, 'D3 24 24 10, 'D3 12 12 10, 'D3 12 12 10 , 'D3 8 8 16, 'D3 4 4 16, 'D1 256, 'D1 256 , 'D1 80, 'D1 80, 'D1 10, 'D1 10] This is a much simpler and more concise description of our network than we can get in Tensor Flow! Let’s examine the ways in which the library uses dependent types to its advantage. The Magic of Dependent Types Describing our network as a type seems like a strange idea if you’ve never used dependent types before. But it gives us a couple great perks! The first major win we get is that it is very easy to generate the starting values of our network. Since it has a specific type, we can let type inference guide us! We don’t need any term level code that is specific to the shape of our network. All we need to do is attach the type signature and call ! randomNetwork randomSimple :: MonadRandom m => m SimpleNetworkrandomSimple = randomNetwork This will give us all the initial values we need, so we can get going! The second (and more important) win is that we can’t build an invalid network! Suppose we try to take our simple network and somehow format it incorrectly. For instance, we could say that instead of the input shape being of size 4, it’s of size 7: type SimpleNetwork = Network ‘[FullyConnected 4 10, Relu, FullyConnected 10 3] ‘[ ‘D1 7, ‘D1 10, ‘D1 10, `D1 3]-- ^^ Notice this 7 This will result in a compile error, since there is a mismatch between the layers. The first layer expects an input of 4, but the first data format is of length 7! Could not deduce (Layer (FullyConnected 4 10) ('D1 7) ('D1 10)) arising from a use of ‘randomNetwork’ from the context: MonadRandom m bound by the type signature for: randomSimple :: MonadRandom m => m SimpleNetwork at src/IrisGrenade.hs:29:1-48 In other words, it notices that the chain from to using a layer is invalid. So it doesn’t let us make this network. The same thing would happen if we made the layers themselves invalid. For instance, we could make the output and input of the two fully-connected layers not match up: D1 7 D1 10 FullyConnected 4 10 -- We changed the second to take 20 as the number of input elements.type SimpleNetwork = Network '[FullyConnected 4 10, Relu, FullyConnected 20 3] '[ 'D1 4, 'D1 10, 'D1 20, 'D1 3] … /Users/jamesbowen/HTensor/src/IrisGrenade.hs:30:16: error: • Could not deduce (Layer (FullyConnected 20 3) ('D1 10) ('D1 3)) arising from a use of ‘randomNetwork’ from the context: MonadRandom m bound by the type signature for: randomSimple :: MonadRandom m => m SimpleNetwork at src/IrisGrenade.hs:29:1-48 So Grenade makes our program much safer by providing compile time guarantees about our network’s validity. Runtime errors due to dimensionality are impossible! Training the Network on Iris Now let’s do a quick run-through of how we actually train this neural network. Readers with a keen eye may have noticed that the we’ve built is the same network we used to train the Iris data set. So we’ll do a training run there, using the following steps: SimpleNetwork Write the network type and generate a random network from it Read our input data into a format that Grenade uses Write a function to run a training iteration. Run it! 1. Write the network type and generate the network So we’ve already done this first step for the most part. We’ll adjust the names a little bit though. Note that I’ll include the imports list as an appendix to the post. Also, the code is on the of my repository in ! [grenade](https://github.com/jhb563/HTensor/tree/grenade) branch Haskell Tensor Flow IrisGrenade.hs type IrisNetwork = Network '[FullyConnected 4 10, Relu, FullyConnected 10 3] '[ 'D1 4, 'D1 10, 'D1 10, 'D1 3] randomIris :: MonadRandom m => m IrisNetworkrandomIris = randomNetwork runIris :: FilePath -> FilePath -> IO ()runIris trainingFile testingFile = do initialNetwork <- randomIris ... 2. Take in our input data We’ll make use of the function we used back when we . Then we'll make a dependent type called , which uses the type. This type is a container for a shape. We want our input data to use for the 4 input features. Then our output data should use for the three possible categories. readIrisFromFile first did Iris IrisRow S S D1 4 D1 3 -- Dependent type on the dimensions of the rowtype IrisRow = (S ('D1 4), S ('D1 3)) If we have malformed data, the types will not match up, so we’ll need to return a to ensure this succeeds. Note that we normalize the data by dividing by 8. This puts all the data between 0 and 1 and makes for better training results. Here's how we parse the data: Maybe parseRecord :: IrisRecord -> Maybe IrisRowparseRecord record = case (input, output) of (Just i, Just o) -> Just (i, o) _ -> Nothing where input = fromStorable $ VS.fromList $ float2Double [ field1 record / 8.0, field2 record / 8.0, field3 record / 8.0, field4 record / 8.0] output = oneHot (fromIntegral $ label record) Then we incorporate these into our main function: runIris :: FilePath -> FilePath -> IO ()runIris trainingFile testingFile = do initialNetwork <- randomIris trainingRecords <- readIrisFromFile trainingFile testRecords <- readIrisFromFile testingFile let trainingData = mapMaybe parseRecord (V.toList trainingRecords) let testData = mapMaybe parseRecord (V.toList testRecords) -- Catch if any were parsed as Nothing if length trainingData /= length trainingRecords || length testData /= length testRecords then putStrLn "Hmmm there were some problems parsing the data" else … 3. Write a function to train the input data This is a multi-step process. First we’ll establish our learning parameters. We’ll also write a function that will allow us to call the function on a particular row element: train learningParams :: LearningParameterslearningParams = LearningParameters 0.01 0.9 0.0005 -- Train the network!trainRow :: LearningParameters -> IrisNetwork -> IrisRow -> IrisNetworktrainRow lp network (input, output) = train lp network input output Next we’ll write two more helper functions that will help us test our results. The first will take the network and a test row. It will transform it into the predicted output and the actual output of the network. The second function will take these outputs and reverse the process to get the label out (0, 1, or 2). oneHot -- Takes a test row, returns predicted output and actual output from the network.testRow :: IrisNetwork -> IrisRow -> (S ('D1 3), S ('D1 3))testRow net (rowInput, predictedOutput) = (predictedOutput, runNet net rowInput) -- Goes from probability output vector to labelgetLabels :: (S ('D1 3), S ('D1 3)) -> (Int, Int)getLabels (S1D predictedLabel, S1D actualOutput) = (maxIndex (extract predictedLabel), maxIndex (extract actualOutput)) Finally we’ll write a function that will take our training data, test data, the network, and an iteration number. It will return the newly trained network, and log some results about how we’re doing. We’ll first take only a sample of our training data and adjust our parameters so that learning gets slower. Then we’ll train the network by folding over the sampled data. run :: [IrisRow] -> [IrisRow] -> IrisNetwork -> Int -> IO IrisNetworkrun trainData testData network iterationNum = do sampledRecords chooseRandomRecords (V.fromList trainData) -- Slowly drop the learning rate let revisedParams = learningParams { learningRate = learningRate learningParams * 0.99 ^ iterationNum} let newNetwork = foldl' (trainRow revisedParams) network sampledRecords .... Then we’ll wrap up the function by looking at our test data, and seeing how much we got right! run :: [IrisRow] -> [IrisRow] -> IrisNetwork -> Int -> IO IrisNetwork run trainData testData network iterationNum = do sampledRecords chooseRandomRecords (V.fromList trainData) -- Slowly drop the learning rate let revisedParams = learningParams { learningRate = learningRate learningParams * 0.99 ^ iterationNum} let newNetwork = foldl' (trainRow revisedParams) network sampledRecords let labelVectors = fmap (testRow newNetwork) testData let labelValues = fmap getLabels labelVectors let total = length labelValues let correctEntries = length $ filter ((==) fst snd) labelValues putStrLn $ "Iteration: " ++ show iterationNum putStrLn $ show correctEntries ++ " correct out of: " ++ show total return newNetwork 4. Run it! We’ll call this now from our main function, iterating 100 times, and we’re done! runIris :: FilePath -> FilePath -> IO ()runIris trainingFile testingFile = do ... if length trainingData /= length trainingRecords || length testData /= length testRecords then putStrLn "Hmmm there were some problems parsing the data" else foldM_ (run trainingData testData) initialNetwork [1..100] Comparing to Tensor Flow So now that we’ve looked at a different library, we can consider how it stacks up against Tensor Flow. So first, the advantages. Grenade’s main advantage is that it provides dependent type facilities. This means it is more difficult to write incorrect programs. The basic networks you build are guaranteed to have the correct dimensionality. Additionally, it does not use a “placeholders” system, so you can avoid those kinds of errors too. This means you’re likely to have fewer runtime bugs using Grenade. Concision is another major strong point. The training code got a bit involved when translating our data into Grenade’s format. But it’s no more complicated than Tensor Flow. When it comes down to the exact definition of the network itself, we do this in only a few lines with Grenade. It’s complicated to understand what those lines mean if you are new to dependent types. But after seeing a few simple examples you should be able to follow the general pattern. Of course, none of this means that Tensor Flow is without its advantages. As we saw a couple weeks ago, it is not too difficult to add very thorough logging to your Tensor Flow program. The Tensor Board application will then give you excellent visualizations of this data. It is somewhat more difficult to get intermediate log results with Grenade. There is not too much transparency (that I have found at least) into the inner values of the network. The network types are composable though. So it is possible to get intermediate steps of your operation. But if you break your network into different types and stitch them together, you will remove some of the concision of the network. Also, Tensor Flow also has a much richer ecosystem of machine learning tools to access. Grenade is still limited to a subset of the most common machine learning layers, like convolution and max pooling. Tensor Flow’s API allows approaches like support vector machines and linear models. So Tensor Flow offers you more options. One question I may explore in a future article would be to compare the performance of the two libraries. My suspicion is that Tensor Flow is faster due to how it gets all its math down to the C-level. But I’m not too familiar yet with HMatrix (which Grenade depends on for its math) and its efficiency. So I could definitely be wrong. Conclusion Grenade provides some truly awesome facilities for building a concise neural network. A Grenade program can demonstrate at compile time that the network is well formed. It also allows an incredibly concise way to define what layers your neural network has. It doesn’t have the Google level support that Tensor Flow does. So it lacks many cool features like logging and visualizations. But it is quite a neat library for its scope. One thing I haven’t mentioned is its mechanics for Generative/Adversarial networks. I’d definitely like to try that out soon! Grenade is a simpler library to incorporate into Stack compared to Tensor Flow. If you want to compare the two, you should check out our so you can install TF and get started! Haskell Tensor Flow guide If you’ve never written a line of Haskell before, never fear! Download our for some free resources to start your Haskell education! Getting Started Checklist Appendix: Compiler Extensions and Imports {-# LANGUAGE DataKinds #-}{-# LANGUAGE BangPatterns #-}{-# LANGUAGE TupleSections #-}{-# LANGUAGE GADTs #-} import Control.Monad (foldM_)import Control.Monad.Random (MonadRandom)import Control.Monad.IO.Class (liftIO)import Data.Foldable (foldl')import Data.Maybe (mapMaybe)import qualified Data.Vector.Storable as VSimport qualified Data.Vector as Vimport GHC.Float (float2Double)import Grenadeimport Grenade.Core.LearningParameters (LearningParameters(..))import Grenade.Core.Shape (fromStorable)import Grenade.Utils.OneHot (oneHot)import Numeric.LinearAlgebra (maxIndex)import Numeric.LinearAlgebra.Static (extract) import Processing (IrisRecord(..), readIrisFromFile, chooseRandomRecords)