DL72

For our purpose now, the functional programming paradigm is particularly valuable. Using functional features, we will write less but more robust code, performing more complex tasks faster.

Let’s talk about another type of matrix-by-matrix multiplication. Sometimes, we need only a coefficient-wise matrix multiplication:

Since our matrix algebra libraries exhaustively use vectorization, we usually aggregate the row data into batches to allow faster operation executions. Consider the following example:

Matrices and tensors are the building blocks of machine learning algorithms. There is no built-in matrix implementation in C++ (and there shouldn’t be one). Fortunately, several mature and excellent linear algebra libraries are available, such as Eigen and Armadillo.

Modern compilers and computer architectures provide an enhanced feature called vectorization. In simple words, vectorization allows independent arithmetic operations to be executed in parallel, using multiple registers. For example, the following for-loop:

In this series, we will learn how to code the must-to-know deep learning algorithms such as convolutions, backpropagation, activation functions, optimizers, deep neural networks, and so on, using only plain and modern C++.

If you require an accessory that you cannot find on our site, please contact us and we will source the component for you.

I have been using Eigen happily for years. Eigen (under the Mozilla Public License 2.0) is header-only and does not depend on any third-party libraries. Therefore, I will use Eigen as the linear algebra backend for this story and beyond.

It turns out that, in modern C++, instead of explicitly using for or while loops, we can rather use functions such as std::transform, std::for_each, std::generate_n, etc., passing functors, lambdas, or even vanilla functions as parameters.

The parameter list and body clauses work like in any regular function. The capture clause specifies the set of external variables addressable in the lambda’s body.

This vectorized version makes the program run four times faster when compared with the original version. Notably, this performance gain happens without impacting the original program’s behavior.

DL83

Artificial Intelligence, Computer Vision & System Architect https://github.com/doleron https://www.linkedin.com/in/doleron/

When I said “pure C++”, it was not entirely true. We will be using a reliable linear algebra library to implement our algorithms.

Vectorization performs an important role in machine learning. For example, batches are often processed in a vectorized way, making a train with large batches run faster than a train using small batches (or no batching).

Each momentum_optimizer(current_grads) call results in a different value even though we pass the same value as the parameter. This happens because we defined the lambda using the keyword mutable.

For the sake of our mission here, C++ included a handy set of common routines in the and headers. As an illustrative example, we can obtain the inner product of two vectors by:

DL75

It is needless to say how relevant machine learning frameworks are for research and industry. Due to their extensibility and flexibility, it is rare to find a project that does not use Google TensorFlow or Meta PyTorch nowadays.

DL70

Once an old language, C++ has drastically evolved in the last decade. One of the main changes is the support of Functional Programming. However, several other improvements were introduced, helping us to develop better, faster, and safer machine learning code.

Usually referred to as mulmat, this operation has a computational complexity of O(N³). Since mulmat is used extensively in machine learning, our algorithms are strongly affected by the size of our matrices.

Machine Vision Direct offers Smart Vision Lights generous 45 day trial period. Contact us if you are interested in trying a light.

Delta

Of course, in coefficient-wise multiplication, the dimension of arguments must match. In the same way, we can add or subtract matrices:

By default, lambdas are immutable objects, i.e., they cannot change the state of objects captured by value. However, we can define a mutable lambda if we want. Consider the following implementation of Momentum:

Need Assistance with specifying or customizing any of our products? Contact us to talk to one of our Automation Engineers!

Even though vectorization is performed by the compiler, operation system, and hardware under the wood, we have to be attentive when coding to allow vectorization:

We will begin our journey in this story by learning modern C++ language features and relevant programming details to code deep learning and machine learning models.

Lambdas are highly useful. We can declare and pass them like old-style functors. For example, we can define an L2 regularization lambda:

This was an introductory talk about how to code deep learning algorithms using modern C++. We covered important aspects of developing high-performance machine learning programs such as functional programming, algebra calculus, and vectorization.

As we already discussed, C++ 11 includes changes in the core of the language to support functional programming. So far, we have seen one of them:

Inverses, transposes, and determinants are fundamental to implementing our models. Another key point is to apply a function to each element of a matrix:

It may seem counter-intuitive to spend time coding machine learning algorithms from scratch without any base framework. However, it is not. Coding the algorithms ourselves provides a clear and solid understanding of how the algorithms work and what the models are really doing.

In some circumstances, following these rules is not easy. Given the complexity and code size, sometimes it is hard to say when a specific part of the code was or wasn’t vectorized by the compiler.

The algorithm header is plenty of useful routines, such as std::transform,std::for_each, std::count, std::unique, std::sort, and so on. Let’s see an illustrative example:

Dl590

Machine Vision Direct is an authorized distributor of Smart Vision Lights, a leading industrial machine vision lighting supplier. When you contact us, our engineers answer the phone and are more than happy to assist you in picking the vision system or accessories that best suit your system. You can even send us application images and details to have us assist in picking out an appropriate vision system.

Tax Exemptions: Please create an account and submit Tax Exemption documentation to Support@MachineVisionDirect.com. After documents have been reviewed and accepted, account will be tagged as tax exempt from all future purchases. Review generally takes 1 hour or less.

Lead Times are estimated based on standard product lead times, but are subject to change. For the latest product lead times, we always recommend contacting us

DL151

C++ is a multi-paradigm programming language, meaning we can use it to create programs using different “styles” such as OOP, procedural, and, recently, functional.

by the compiler. The trick is that the instruction A[i + 1] = A[i + 1] + B[i + 1] runs at the same time as the instruction A[i] = A[i] + B[i]. This is possible because the two instructions are independent of each other, and the underlying hardware has duplicated resources, that is, two execution units.

This lambda consists of 3 clauses: a capture list ([&x, &y] ), a parameter list (const std::function &comparator), and the body (the code between the curly braces{...}).

Instead of performing six inner products between each of the sixXivectors and one Vvector to obtain six outputs Y0, Y1, etc, we can stack the input vectors to mount a matrix M with six rows and run it once using a single mulmat multiplication Y = M*V .

Note that compare is not the lambda name but the name of a variable to which the lambda is assigned. Indeed, lambdas are anonymous objects.

DL73

Your payment information is processed securely. We do not store credit card details nor have access to your credit card information.

Some relevant programming topics of real-world ML projects were not covered here, like GPU programming or distributed training. We shall talk about these subjects in a future story.

Here, we use std::function, std::less, std::less_equal, std::greater, andstd::greater_equal as an example of polymorphic calls in action without using pointers.

As a rule of thumb, the more streamlined and straightforward the code, the more prone to be vectorized it is. Therefore, using standard features of , algorithm , functional, and STL containers indicate code that is more likely to be vectorized.