Tweet

Not Hotdog - Salesforce Einstein Edition

Introduction

Anyone who is a fan of HBO’s Silicon Valley show will be familiar with Not Hotdog, Jian Yang’s app that determines whether an item of food is a hotdog or not. In a wonderful example of fiction made fact, the show have released iOS and Android applications in real life - you can read about how they did this on their medium post. Around this time I was working through the Build a Cat Rescue App that Recognises Cat Breeds Trailhead Project, which uses Einstein Vision to determine the breed of cat from an image, and it struck me that I could use this technology to develop a Salesforce version of Not Hotdog.

Building blocks

Trailhead Playground

As I’d already set up Einstein Vision and connected it to my Trailhead Playground, I decided to build on top of that rather than create a new developer edition. 

Einstein Vision Apex Wrappers

A key aspect of the project is the salesforce-einstein-vision-apex repository - Apex wrappers for Einstein Vision produced by Developer Evangelist René Winkelmeyer. The project somewhat glosses over these, but they provide a really nice mechanism to create and train an Einstein Vision dataset and then use that for predictions. It takes away pretty much all the heavy lifting, so thanks René. 

Public Access Community

Let’s be honest, there was no way I was going to build a full-fledged app for this. I did consider building an unmanaged package and including the images I used to train the dataset, but it seemed a bit crazy to have everyone creating and training their own dataset for the same purpose. Given my reach in the Salesforce community this could literally result in tens of duplicate datasets :)

I therefore decided to expose this as an unauthenticated page on a Salesforce community. I had the option of using a Site but I also wanted to play around with unauthenticated access to Lightning Components and the docs say to use a community. 

Putting it all together

I had to make one change to the Einstein Vision Apex Wrappers - I couldn’t get the guest user to be able to access the Salesforce File containing the Einstein Vision key, so I just hardcoded it into the EinsteinVision_PredictionService class. Evil I know, but this is hardly going into production any time soon.

I then created a dataset named ‘nothotdog’ and trained it via a zip file of images. The zip file is organised into a directory per label - in my case there were two directories - ‘Hot Dog’ and ‘Not Hot Dog’.

I then added the following method to the EinsteinVision_Admin class, to match a supplied image in base64 form against the dataset.

@AuraEnabled
public static String GetHotDogPredictionKAB(String base64) {
    String hdLabel='Unable to match hotdog';
    Blob fileBlob = EncodingUtil.base64Decode(base64);
    EinsteinVision_PredictionService service = new EinsteinVision_PredictionService();
    EinsteinVision_Dataset[] datasets = service.getDatasets();
    for (EinsteinVision_Dataset dataset : datasets) {
        if (dataset.Name.equals('nothotdog')) {
            EinsteinVision_Model[] models = service.getModels(dataset);
            EinsteinVision_Model model = models.get(0);
            EinsteinVision_PredictionResult result = service.predictBlob(model.modelId, fileBlob, '');
            EinsteinVision_Probability probability = result.probabilities.get(0);
        }
    }
        
    return hdLabel;
}

Next I needed a lightning component that would allow me to upload a file and send it back to the server, to execute the method from above. However, I also wanted this to work from a mobile device as file inputs on the latest Android and iOS allow you to take a picture and use that. The problem with this is that the image files are pretty huge, so I also needed a way to scale them down before submitting them. Luckily this can be achieved by drawing the image to an HTML5 canvas element scaled to the appropriate size.

Unfortunately this threw up another problem, in that when the Locker Service is enabled you don’t have an image element that can be drawn on a canvas, you have a secure element instead. There is no workaround to this so I had to drop the API version of my component down to 39. I guess one day the Locker Service will be finished and everything will work fine.

There’s a fair bit of code in the NotHotdog Lightning Component bundle so rather than making this the world’s longest post you can view it at this gist.

Next, I needed an app to surface the bundle through a Visualforce page. These are pretty simple, the only change to the usual way this is done is to implement the interface ltng:allowGuestAccess:

<aura:application access="GLOBAL" extends="ltng:outApp"
    implements="ltng:allowGuestAccess">

    <aura:dependency resource="c:NotHotDog"/>
</aura:application>

Finally, the Visualforce page that is accessible via the community:

<apex:page docType="html-5.0" sidebar="false" showHeader="false" standardStylesheets="false"
           cache="false" applyHtmlTag="false">
    <html>
        <head>
            <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no;" />
        </head>
        <body>
            <apex:includeLightning />
            <script>
            $Lightning.use("c:NotHotDogApp",
                           function() {
                               $Lightning.createComponent(
                                   "c:NotHotDog",
                                   { },
                                   "lightning",
                                   function(cmp) {
                                   }
                               );
                           }
                          );
            </script>
            <div id="lightning" />
        </body>
    </html>
</apex:page>

Yes we’ve got a video

Here’s a video of the app doing it’s thing - first recognising a hotdog and then correctly determining that the BrightGen head office building is not a hotdog. What a time to be alive.

 

 

It’s not bullet proof

The HBO team trained their app with hundreds of thousands of images, I just did a couple of hundred because this isn’t my day job! It’s pretty good on obvious hotdog images, but not so much when you take photos. Your mileage may vary. Also, take photos on a phone in landscape mode as most of them rotate it.

Try it out yourself

If you’d like to enjoy the majesty of this application on your own machine:

• Navigate to the short link : http://bobbuzz.me.uk/nothotdog or
• Scan the QR code below:

Meetup

if you’re in London on Aug 2nd 2017, we’ll have a talk on Einstein Vision at our developer meetup. Sign up at : 

    https://www.meetup.com/LondonSalesforceDevelopers/events/237321315/

Related Information

• Build a Cat Rescue App that Recognises Cat Breeds (Trailhead Project)
• salesforce-einstein-vision-apex (github repository)
• Einstein API Documentation (well worth a look to understand what the apex wrappers are doing under the hood)
• How HBO’s Silicon Valley built “Not Hotdog” with mobile TensorFlow, Keras & React Native (Medium post)

 

 Follow @bob_buzzard