Qeexo

TRY
BACK TO PRESS

Introducing Qeexo Model Converter

Gilbert Tsang, Director of Product Management 29 April 2021

Our latest API service for fitting your existing ML models onto an embedded target as small as a Cortex-M0+! 

Qeexo AutoML offers end-to-end machine learning with no coding required. While this SaaS product presents a wholistic user experience, we understand that machine learning (ML) practitioners working in the tinyML space may want to use their preexisting models that they’ve already spent a lot of time and efforts to finetune. To these folks working on tinyML applications, fitting the models onto embedded hardware with constrained resources is the final step before they can test their models on the embedded Edge device. However, this step requires a specialized set of embedded knowledge that may be outside of a typical ML engineer’s repertoire. 

Qeexo addresses this pain point by offering an API-based model converter service. At launch, the Qeexo Model Converter currently converts tree-based models (Random ForestXGBoostGradient Boosting Machine) for Arm Cortex-M0+ to Cortex-M4 embedded targets. 

Let’s dive into more details! 

Qeexo’s approach to tree-based model conversion 

While there are dozens of different machine learning algorithms in use, both open-source and proprietary model conversion solutions largely focus on converting neural network (NN) models. From our experience, tree-based models often outperform NN models in tinyML applications because they require less training data, have lower latency, are smaller in size, and do not need a significant amount of RAM during inference. Our team at Qeexo first developed proprietary methods to convert tree-based models for embedded devices for our internal use, since we were unable to find comparable solutions on the market. 

Qeexo Model Converter contains patent-pending quantization technologies, as explained by Dr. Schradin, Principal ML Engineer at Qeexo, in this tinyML talk. Our model converter utilizes intelligent pruning and quantization technologies that enable these tree-based ensemble models to have a low memory footprint without compromising classification performance. (Note that the tree-based models can be pruned post-training, while NN models usually need to be re-trained after compression.) 

This conversion process outputs optimized object code with metadata, which can easily be integrated into Arm Cortex-M0+ to Cortex-M4 embedded platforms. 

API-based Qeexo Model Converter 

Our model converter can be accessed through a RESTful API that can be called from a wide range of programming languages. We chose the ONNX format as the input to the Qeexo Model Converter, which enables us to support both scikit-learn models as well as XGBoost tree models. We also feel that this open standard offers exceptional interoperability among different workflow architectures.  

Figure 1: Qeexo Model Converter Workflow

Perhaps the coolest and most useful feature of the Qeexo Model Converter is the ability to limit models to a given size – when provided with a maximum size, our converter will try its best to reduce the input model to this desired size by applying a tree-pruning technique. This tree-pruning technique is different from and in addition to the quantization feature. When enabled, the data arrays are stored in smaller integer types, resulting in further reduction in model size. 

For more detailed instructions on how to use our API, please refer to the user guide for example code. 

Come try it out! 

We hope that you are ready to sign up for a Qeexo account (the same one that you use to log into Qeexo AutoML) and subscribe to the Qeexo Model Converter service (comes with a 30-day free trial)!  

As far as future roadmap is concerned, we are considering extending support to other tree-based models as well as neural network models. Our goal is to provide model conversion as a service so that ML practitioners working in tinyML are free to try different algorithms for their embedded projects, just like the way Qeexo AutoML offers more than a dozen algorithms.  

We would love to get your valuable feedback in order to further improve our model conversion service and build in additional features. Please email us at modelconverter@qeexo.com

BACK TO PRESS

Building effective IoT applications with tinyML and automated machine learning

Embedded 27 January 2021

IoT enables continuous monitoring of environments and machines using tiny sensors. Advances in sensor technologies, microcontrollers, and communication protocols made mass production of IoT platforms, with many connectivity options, possible at affordable prices. Due to the low cost of IoT hardware, sensors are being deployed on a large scale at public places, residentials, and on machines.

Read the full article here: https://www.embedded.com/building-effective-iot-applications-with-tinyml-and-automated-machine-learning/

BACK TO PRESS

The insideBIGDATA IMPACT 50 List for Q1 2021

insideBIGDATA 05 January 2021

The team here at insideBIGDATA is deeply entrenched in following the big data ecosystem of companies from around the globe. We’re in close contact with most of the firms making waves in the technology areas of big data, data science, machine learning, AI and deep learning. Our in-box is filled each day with new announcements, commentaries, and insights about what’s driving the success of our industry so we’re in a unique position to publish our quarterly IMPACT 50 List of the most important movers and shakers in our industry. These companies have proven their relevance by the way they’re impacting the enterprise through leading edge products and services. We’re happy to publish this evolving list of the industry’s most impactful companies!

Read the full article here: https://insidebigdata.com/2021/01/05/the-insidebigdata-impact-50-list-for-q1-2021/

BACK TO PRESS

Big Data Industry Predictions for 2021

InsideBigData 29 December 2020

2020 has been year for the ages, with so many domestic and global challenges. But the big data industry has significant inertia moving into 2021. In order to give our valued readers a pulse on important new trends leading into next year, we here at insideBIGDATA heard from all our friends across the vendor ecosystem to get their insights, reflections and predictions for what may be coming. We were very encouraged to hear such exciting perspectives. Even if only half actually come true, Big Data in the next year is destined to be quite an exciting ride. Enjoy!

Read the full article here: https://insidebigdata.com/2020/12/21/big-data-industry-predictions-for-2021/

BACK TO PRESS

LETU Machine Learning Contest Video

KLTV 12 November 2020

Click HERE to view the video

Full URL to video source: https://www.kltv.com/video/2020/11/06/machine-learning-contest/

BACK TO PRESS

Can a piece of drywall be smart? Bringing machine learning to everyday objects with TinyML

Diginomica 11 November 2020

So-called smart devices like Amazon Echo and Google Nest made early headway into our homes. But will devices as small as a vibration sensor soon outsmart an Echo? Here’s a look under the hood of “TinyML.”

Read the full article at: https://diginomica.com/can-piece-drywall-be-smart-bringing-machine-learning-everyday-objects-tinyml

BACK TO PRESS

LeTourneau University students design artificial intelligence projects for contest

Longview News-Journal 09 November 2020

Since mid-September, 10 teams of LeTourneau University engineering students have been working on projects involving artificial intelligence to enter into a contest. Winners of that contest were announced Friday in the lobby of the Glaske Engineering Center after demonstrations from students.

The teams were challenged by contest sponsors Qeexo, the maker of the machine learning platform AutoML, and Arduino, an open-source electronics hardware and software platform, to provide solutions for real world problems using embedded machine learning. Students’ projects include a device that monitors hand movements to allow it to be almost unbeatable in a game of rock, paper, scissors to a another device that helps fly fishermen perfect their cast.

Link to the full article: https://www.news-journal.com/news/local/letourneau-university-students-design-artificial-intelligence-projects-for-contest/article_c936dc60-2088-11eb-930c-bb89584d78c8.html

BACK TO PRESS

Live Classification Analysis

Sidharth Gulati, Dr. Rajen Bhatt 03 November 2020

Qeexo AutoML enables machine learning application developers to do analysis of different performance metrics for their use-cases and equip them to make decisions regarding ML models like tweaking some training parameters, adding more data etc. based on those real-time test data metrics. In this article, we will discuss in detail regarding live classification analysis module.

Figure 1: Live Classification Analysis

Once the user clicks on Live Classification Analysis for a particular model, they will be directed to the Live Classification Analysis module that would resemble below screenshot.

Figure 2: Live Classification Analysis

In this module we won’t be discussing Sensitivity analysis. To refer to details regarding sensitivity analysis, please read this blog.

Figure 3: Confusion Matrix

For the purpose of this blog, we will use a use-case which aims to classify a few musical air gestures: Drums, Violin and Background. These datasets can be found here.

Live Data Collection

Qeexo AutoML supports live data collection module which can be used to collect data to do analysis on. Data

collection requires a Data collection library to be pushed to the respective hardware. A user can push the library by clicking the “Push To Hardware” button shown below.

Figure 4: Push to Hardware Screen

Once, they click the button and the library flashing is successful, the user will be able to record the data for trained classes in the model for analysis purpose. The user can select any number of seconds of data to do the analysis on. For this particular use-case, we have 3 Classes: Drums, Background and Violin as shown below.

Figure 5: Data Recording Input Screen

Once the user clicks “Record”, they will be redirected to Data Collection page as shown below. This module is same as the Data Collection module which is used to collect training data.

Figure 6: Recording Screen

As the user collects data for respective classes, they will be able to able to see the data in tabular format shown below. They can see the dataset information, delete data and re-record based on their preference.

Figure 7: Dataset Collection

Once, the user has collected the data, they can select whichever data they want to do analysis on by selecting the checkbox as shown above. Once, the user has selected atleast 1 dataset, they will see the Analyze button is activated and as we say, with Qeexo AutoML, “a click is all you need to do Machine Learning”, they will be able to analyze different performance metrics!

Figure 8: Analyze

Performance Metrics

Qeexo AutoML supports 5 different types of performance metrics listed below:

  1. Confusion Matrix: Represents True Labels and Predicted Labels in square matrix. Diagonal (upper left to lower right) elements indicates instances correctly classified. Off-diagonal elements indicate instances mis-classified. Summing instances over each row should sum to total instances for the respective class.
  2. F-1 Score: Measures the 1st harmonic mean of Precision and Recall. Computed as 2 * (Precision * Recall)/(Precision + Recall). Precision measures out of all the samples detected of a given class, how many are relevant. Recall measures out of all the relevant samples of a given class, how many are detected.
  3. Matthews Correlation Coefficient: Measure of discriminative power for binary classifiers. In the multi-class classification case, it quantifies which combinations of classes are the least distinguished by the model. The values can range between -1 and 1, although most often in AutoML the values will be between 0 and 1. A value of 0 means that the model is not able to distinguish between the given pair of classes at all, and a value of 1 means that the model can perfectly make this distinction.
  4. ROC Curve: Plots the False Positive Rate (FPR, x-axis) vs. True Positive Rate (TPR, y-axis) for each class in the classification problem. The dotted line indicates flip-of-the-coin performance where the model has no discriminative ability to distinguish among the classes. The greater the area under the curve (AUC), the better the model.
  5. Kernel Density Estimation plots: This will result in n plots, where n = Number of trained classes. This plot shows the estimated probability density function for each class vs rest of the classes.

For the use case of this blog, please find respective metrics below:

Confusion Matrix

Figure 9: Confusion Matrix

ROC Curve

Figure 10: ROC Curve

Matthews Correlation Coefficient

Figure 11: Matthews Correlation Coefficient

F-1 Score

Figure 12: F-1 Score

Kernel Density Estimation (KDE)

Figure 13: KDE for Background vs Rest of the labels
Figure 14: KDE for Drums vs Rest of the labels
Figure 15: KDE for Violin vs Rest of the labels

With these performance metrics, a user can determine how “well” the model is performing on test data or in live classification scenario. With the help of this module, a user can decide different aspects of a ML pipeline like whether to retrain a model with different parameters, whether more data will help improving the performance or different sensitivities for different classes should be considered. In a nutshell, Live Classification Analysis enables the user to take more control over ML model development cycle based on performance analysis on test data.

BACK TO PRESS

Sensitivity Analysis with Qeexo AutoML

Qifan He, Dr. Rajen Bhatt 29 October 2020

Introduction


For machine learning models, Sensitivity parameter reflects on how sensitive the model is for classes under consideration. Sensitivity Analysis is generally performed before deployment of ML models in the real world application. The primary objective of the Sensitivity Analysis is to make ML model lean more towards certain class(es) than the other(s). Often the sensitivity analysis is also related to the study of the tolerance for misclassifying instances of certain class(es) against the other(s). For example, consider the machine learning model designed to detect faults in the industrial equipment. Generally, an operator want to always make sure that defects, if any, are detected almost always. In this case, an operator is OK (even though it is not ideal) if some non-defects are being recognized as defects. Because the cost of defects not being recognized as defects is very high as this may damage the equipment permanently. While there are also costs of classifying non-defects as defects, these costs are comparatively very less and can be filtered manually as false alarms. In general, ML algorithms should try to reduce the false alarms as well. In this blog, we will discuss how to perform sensitivity analysis on Qeexo AutoML platform.

Sensitivity Analysis

Qeexo AutoML performs the sensitivity analysis using Class Weights. For the classification problem having C, C >= 2, number of classes, class weights is a C-dimensional array of integers > 0. During the model training phase, Qeexo AutoML assigns the weight of 1 to each classes, i.e., the initial (or default) weight vector is C-dimensional array of 1’s which can be represented as {w_1, w_2, ..., w_C}. This results in the initial sensitivity value of 1/C for each class represented as s_1, s_2, ..., s_C such that

    \[ \sum _{i=1}^{C}s_i = 1\ \]

For example, for binary classification problem, i.e., 2-class classification problem, the default sensitivity array is s_1, s_2 = 0.5, 0.5. If we start lowering one of the numbers, the model becomes more sensitive to that particular class. Lowering the sensitivity number of particular class is equivalent to increasing the weight of that class. All the model performance metrics such as Confusion matrix, Learning Curves, ROC curves, F-1 Score, and MCC are all computed with the default sensitivity value 1/C and class weights 1.

After training models on the Qeexo AutoML platform, you will be guided towards the Models page. Here you can see the details of each model and perform the live classification. You can go to the Live Classification Analysis to analyze the sensitivity of each class and update their influence on the model performance.

Figure 1: model details

When you click into Live Classification Analysis icon, you can see the following page.

Figure 2: Live Classification Analysis

In the first tab, you can see the description of the model, such as the classes used in training and the date it has been created. The second section Compiled History will save the history of the weights you have tried; when you click this for the very first time, it will show the default weights of 1 for each class. It also allows you to select and delete any of the weights combination in your history. The selected weights will be updated on your device once you click Selected on this page and push the library to the device with the button in the Live-data Collection tab or on the Model page.

The bigger the weight is, the more sensitive the model is to that class. In other words, the model is more likely to output class with the higher weight. Even though you can assign a weight to each class, only the relative differences between the class weights matter. That is, for three-class classification, weights {1,1,1} will have the same effect as {3, 3, 3} because this simply means each class has equal weights.

In the third tab, you can try different combinations of weights and see the simulation of their effects on model performance. The model performance is shown through two metrics. One is the bar chart showing the the accuracy of each class with the chosen weight combination. The second one is through the confusion matrix. The y-axis of this table is True Label, the x-axis is Predicted Label. The values on diagonal from top left to bottom right of this table is where predicted label matches true label. The perfect performance shown by confusion matrix should have zeros everywhere except the diagonal cells.

The last part of this page offers a chance to collect some new testing data and evaluate the model with the selected weights on the testing data.

Some Examples

Let us first take a look at a binary classifier example. For binary classifier, the default classification rule is the following:

    \[ y= \begin{dcases} class_1, & \text{if} P(class_1) > 0.5\\ class_0, & \text{otherwise} \end{dcases} \]

However, in reality, in order to make the classifier more sensitive towards class-0 using this model, we want to make the following classification rule:

    \[ y= \begin{dcases} class_1, & \text{if} P(class_1) > 0.75\\ class_0, & \text{otherwise} \end{dcases} \]

The new classification rule is stricter for class-1 and relaxed for class-0. This may be the better model compared to the default model because we may want to detect class-1 only if probability assignment of class-1 is highly confident, e.g. >=0.75, otherwise we want to classify the incoming signal (or pattern) into class-0. With this classification rule, the model remains the same but becomes more sensitive to one class over the other(s). In order to achieve this classification rule, the weights are computed as given below:

    \begin{align*} weight_0= 0.5/0.25 = 2 \\ weight_1 = 0.5/0.75 = 0.67 \end{align*}

For example, the models assigned the probabilities to each class as {0.4, 0.6}. With new weights, the weighted probabilities are:

    \begin{align*} weighted\_probability= [2, 0.67] * [0.4, 0.6] = [0.80, 0.40] \end{align*}

Now we need to compare weighted_probability with the default thresholds {0.5,0.5}, that results in the classification decision class-0. With the concept of weights for classes, we have achieved the effect as if the sensitivities are {0.25, 0.75} for each class. Please note, without weights, this signal would have classified to class-1. However, with relaxed sensitivity value for class-0 and stricter sensitivity value for class-1, we get the classification outcome as class-0. Note that the sum of probabilities doesn’t equal to 1. We can also normalize the probabilities and get the same prediction.

    \begin{align*} normalized\_weighted\_probability= [0.80/(0.8 + 0.40), 0.40/(0.80 + 0.40)] = [0.666, 0.333] \end{align*}

This weighted probability compute generalize very well with multi-class classification with each class having its own threshold.

Conclusion

For real-world applications, finding the right weights for each class is a matter of trial-and-error or some predefined human knowledge. Qeexo AutoML offers very efficient method to test with different class weights, quickly check the classification performance, and then push the newly determined class weights in order to perform the live testing.

BACK TO PRESS

The insideBIGDATA IMPACT 50 List for Q4 2020

insideBIGDATA 13 October 2020

The team here at insideBIGDATA is deeply entrenched in following the big data ecosystem of companies from around the globe. We’re in close contact with most of the firms making waves in the technology areas of big data, data science, machine learning, AI and deep learning. Our in-box is filled each day with new announcements, commentaries, and insights about what’s driving the success of our industry so we’re in a unique position to publish our quarterly IMPACT 50 List of the most important movers and shakers in our industry. These companies have proven their relevance by the way they’re impacting the enterprise through leading edge products and services. We’re happy to publish this evolving list of the industry’s most impactful companies!

Read the full article here: https://insidebigdata.com/2020/10/13/the-insidebigdata-impact-50-list-for-q4-2020/