Showing posts with label SalesforceDX. Show all posts
Showing posts with label SalesforceDX. Show all posts

Sunday, 14 January 2024

Scratch Org Snapshots in Spring '24

Note: This feature is in beta in Spring '24. Like all other betas, this functionality may never go GA and may disappear at any time. Caveat emptor.

Image generated by DALL-E 3 based on a prompt by Bob Buzzard

Introduction

The Spring '24 release of Salesforce moves the scratch org snapshot functionality into beta. I've been waiting to get my hands on this and so far it hasn't disappointed.

Whenever we get new features of this nature, I like to reflect on how far we've come in Salesforce land. In this case I was testing with the codebase of our BrightMEDIA accelerator, and when we started building this in mid-2014 (aka nearly a decade ago), we typically allowed a week to get a new developer set up. We had to spin up a Developer Edition, raise a bunch of tickets to get various features enabled and increase the Apex character limit, install a number of packages, carry out a number of manual setup steps,  deploy the code and assign permission sets. For whatever reason, no two Developer Editions appeared to have the same setup, so typically the deployment was an iterative process where we discovered what was missing or off instead of on by default. Then they'd go through and set up some standing data to be able to work in the org. 

Fast forward to the end of 2023 and I have a node script that creates a scratch org, installs the packages, deploys the code, loads the standing data and produces a ready to go development environment in around 30 minutes. I'm always interested in speeding things up though!

Creating a Snapshot

Thanks to a pre-release environment that I've also had for a decade, I have a pre-release dev hub which meant I could enable the beta before the Spring '24 release goes live. Then I assigned myself the appropriate object permissions for Org Snapshots and I was ready to create. 

I set up my scratch org using my existing script, which creates an org with the following applied:

  • Four managed packages
  • Approximately 9,000 metadata components
  • Approximately 2,000 records
Creating a snapshot of this org took 11 minutes, which I must admit was quite a bit faster than I was expecting.

Using the Snapshot


This started off with a bit of a challenge, in that attempting to use the snapshot kept giving the error that the snapshot wasn't Active, but listing the Dev Hub snapshots showed that it was indeed Active. I spent a while searching through the CLI Github issues list and the snapshot pilot Trailblazer group, but it seemed like I was the lucky one who got to experience this first. This was quite soon after the pre-release had gone live, so I figured it might be a simple bug and played the waiting game.

About 7 hours later my masterful inactivity was rewarded, as my snapshot sprang to life and I was able to run the commend to create an org from it. In fairness, it might have started working 10 minutes later, but it was around 7 hours later I had the time to try it out again.

The even better news was that creating a scratch org from the snapshot took 6 minutes - an 80% saving on the 30 minute creation time for my script. The org was flawless too - all the metadata and data was there.

The End of Sandboxes?


So does this mean that we can all create scratch org snapshots rather than sandboxes going forward? They even contain data, so maybe we can do away with full or partial copy sandboxes too.  I don't think so, for a few reasons.

Lifespan


Scratch orgs and org snapshots, have a 30 day lifespan. From a developer perspective this is fine - we treat these orgs and disposable and typically create a fresh one when we start a new piece of work. That isn't necessarily the case for orgs used for training, QA, integration testing or testing against a new release. It's particularly unsuitable for pre-production environments which mirror production - imagine having to recreate all your test integrations at the start of every month!

Storage


Scratch orgs and org snapshots are limited to 200Mb of data. Again, probably fine for many development tasks, but again likely to be too small for training, pre-production and test environments that are indicative of production. 

Licenses


Sandboxes replicate your production org licenses, so all of your users can have access. Scratch orgs are a much more restrictively licensed, usually somewhere between 1 and 10 seats per feature. When we were adding community (now Experience Cloud) features to BrightMEDIA, we had the princely sum of 1 partner community license available in our scratch orgs - you'd have to be quite brave to promote to production with that kind of limitation on your testing!

Completeness of Version Controlled Metadata


This is where developer/developer pro sandboxes will retain their usefulness once scratch org snapshots are live. Some organisations with large, mature Salesforce orgs won't have all their metadata in version control, because why would they invest the time and money to do that when they don't need to. They'll likely have Apex, flows, lightning components, and maybe some second generation packages in version control, but things like sharing rules, report and dashboard folders, duplicate rules that are managed by administrators probably won't. Yes this is a sweeping generalisation, but you get the general idea. Being able to create a guaranteed replication of production to work in will be an important capability for years to to come in my view.  That said, they'll probably become less used as time goes on and maybe scratch org snapshots get longer lifespans.

So not a sandbox killer, but that was never the intention. For those of us with a very source-centric development approach however, this is another great addition to the developer toolbelt.

More Information



Saturday, 29 August 2020

Adding IP Ranges from a Salesforce CLI Plug-in


Introduction

I know, another week, another post about a CLI plug-in. I really didn't intend this to be the case, but I recently submitted a (second generation) managed package for security review and had to open up additional IP addresses to allow the security team access to a test org. I'm likely to submit more managed packages for review in the future, so I wanted to find a scriptable way of doing this. 

It turned out to be a little more interesting than I'd originally expected, for a couple of reasons.

It Requires the Metadata API


To add an IP range, you have to deploy security settings via the metadata API. I have done this before, to enable/disable parallel Apex unit tests, but this was a little different. If I create a security settings file with the new range(s) and deploy them, per the Metadata API Docs, all existing IP ranges will be turned off:

    To add an IP range, deploy all existing IP ranges, including the one you
    want to add. Otherwise, the existing IP ranges are replaced with the ones
    you deploy. 

Definitely not what I want!

It Requires a Retrieve and Deploy


In order to get the existing IP addresses, I have to carry out a metadata retrieve of the security settings from the Salesforce org, add the ranges, then deploy them. No problem here, I can simply use the retrieve method on the metadata class that I'm already using to deploy. Weirdly, unlike the deploy function, the retrieve function doesn't return a promise, instead it expected me to supply a callback. I couldn't face returning to callback hell after the heaven of async/await in my plug-in development, so I used the Node Util.Promisify function that turns it into a function that returns a promise. Very cool.

const asyncRetrieve = promisify(conn.metadata.retrieve);
const retrieveCheck = await asyncRetrieve.call(...);
The other interesting aspect is that I get the settings back in XML format, but I want a JavaScript object to manipulate, which I then need to turn back into XML to deploy.

To turn XML into JavaScript I use fast-xml-parser, as this has stood me in good stead with my Org Documentor. To get at the NetworkAccess element:

import { parse} from 'fast-xml-parser';

  ...

const settings = readFileSync(join(tmpDir, 'settings', 'Security.settings'), 'utf-8');
const md = parse(settings);

let networkAccess = md.SecuritySettings.networkAccess;

Once I've added my new ranges, I convert back to XML using xml2js:

import { Builder } from 'xml2js';

  ...

const builder = new Builder(
   {renderOpts:
      {pretty: true,
       indent: '    ',
       newline: '\n'},
    stringify: {
       attValue(str) {
           return str.replace(/&/g, '&')
                     .replace(/"/g, '"')
                     .replace(/'/g, ''');
       }
    },
    xmldec: {
        version: '1.0', encoding: 'UTF-8'
    }
});

const xml = builder.buildObject(md);

The Plug-In


This is available as a new command on the bbsfdx plug-in - if you have it installed already, just run 

sfdx plugins: update 

to update to version 1.4

If you don't have it installed already (I won't lie, that hurts) you can run:

sfdx plugins:install bbsfdx
 
and to add one or more ranges: 

sfdx bb:iprange:add -r 192.168.2.1,192.168.2.4:192.168.2.255 -u <user>

The -r switch defines the range(s) - comma separate them. For a range, separate the start and end addresses with a colon. For a single address, just skip the colon and the end address.

Related Posts


Saturday, 15 August 2020

Org Documentor and AuraEnabled Classes


Introduction

In the Winter 21 release of Salesforce, access to AuraEnabled methods in Apex classes will be restricted to users with profiles or permission sets that grant access to those classes. If you've been working in a sandbox recently you've probably encountered this already, as the critical update for this was enabled on August 8th.

Figuring out who (if anyone) has access to classes isn't straightforward, as you have to trawl through the profiles and permission sets and check each one. Salesforce have created an Aura Enabled Scanner application that can be installed via an unlocked package, which checks packaged and unpackaged code, but it does require that you login to Salesforce each time you need to check things.

This seemed like a good candidate for my Org Documentor Salesforce CLI Plug-In - something that can be run on a schedule, check that any new classes used as controllers for Aura or Lightning Web Components are accessible, and show which profiles/permission sets have access.


Configuration

To add processing for AuraEnabled classes, an additional stanza is required in the configuration file passed to the bbdoc command - here's what it looks like in my example repo:

"auraenabled": {
    "name": "auraenabled",
    "description": "AuraEnabled Class Access", 
    "subdirectory": ".",
    "image": "images/auraenabled.png",
    "suffix":"object",
    "groups": {
        "other": {
            "name":"other", 
            "title":"AuraEnabled Components",
            "description": "All components with Apex controllers"
        }
    }
}

As with other metadata types, you can specify multiple groups to slice up your components into functional areas - I've lumped them all into one group as I only have one component!


Output

The index (home) page for the org report displays a new card for the AuraEnabled metadata:


An error badge is displayed if there are one or more classes used as controllers for components that aren't accessible from any profile or permission set. 

Clicking in to the details shows the groups and errors:


and clicking into the group shows the detail for each component, with classes that aren't accessible highlighted as errors:



Note that if a Lightning Web Component accesses multiple Apex classes, there will be a row for each class with the same component name in the report, as shown above.


Processing

I was quite pleased by how little code I had to write to add support for this:

  • A classes map structure is created in memory, profilesAndPermSetsByClassname, where each entry has the Apex class name as a key and a value object containing lists of permission sets and profiles with access to the class. This is generated by loading all of the profile and permission set metadata and iterating the ClassAccess sections. 

  • The component groups are iterated, and for each entry in the group:

    • If it is an Aura Component, the controller attribute from the <aura:component/> tag is extracted

    • If it is a Lighting Web Component (which can access multiple Apex classes), @salesforce/apex/ lines are identified and the Apex class name extracted. 

      For each class:

      • The entry for the class in the profilesAndPermSetsByClassname map is extracted. If this is null or both of the profile and permission set lists are empty, the classname is added to the error collection displayed on the group page.

      • A row is added to the JavaScript object backing the group page showing the profiles/permission sets that have access, adding the error highlight colour if there are none.


Plug-In

The updated bbddoc Salesforce CLI plug-in providing AuraEnabled support can be found on NPM, and the source code is available in the Github repository. 


Why not use the AuraEnabledScanner?

You absolutely should - I do, to configure access. This works with the AuraEnabledScanner rather than replacing it. As development continues, new classes and components are added to your codebase. The Org Documentor flags up any that don't have appropriate access, and you can then login to an org that your metadata is deployed to, run the scanner and fix up the access. 

The scanner also handles managed classes which the Org Documentor doesn't, as it works against your metadata on disk rather than everything installed in a specific org instance.


Related Posts

Saturday, 1 August 2020

Visual Studio Codespaces - the tech behind Code Builder

Visual Studio Codespaces - the tech behind Code Builder


Salesforce launched Code Builder in June 2020, just before the 2020 TrailheaDX virtual conference, and it's fair to say there was some excitement around this. Those expecting a Developer Console Mk 2 must have been delighted to see a full-featured IDE that can run in the browser.  Even better, we confirmed with the product team that as the IDE is effectively running in a virtual machine, any required extensions and Salesforce CLI plug-ins can be installed, so you can recreate your local environment with all your favourite features.

I was certainly excited about this and reached out in a number of directions to try to get on the pilot, all of which sadly failed - I needed to be nominated by an Account Executive, and experience has taught me that at our number of licenses our AE typically doesn't even know that pilots are available, much less how to nominate me for one.

Undaunted, I took a different approach and started looking at the technology that powers Code Builder - Visual Studio Codespaces.  

Visual Studio Codespaces

Formerly known as Visual Studio Online, the description from official Microsoft site sums it up very well:

Cloud-hosted dev environments accessible from anywhere

There's some signup and setup required to start using Codespaces, as detailed below.

Sign up for Azure

A Microsoft account and Azure subscription are needed, as Azure hosts the virtual environments (machines). I chose the free subscription, which gives me 12 months free Linux virtual machine access. Compute hours are limited, but in all likelihood I'll use this rarely and then assuming it doesn't require me to spend a bunch of money, little or never once Code Builder is available.

VS Code Extensions

I then had to install a couple of extensions :
  • Azure Account, which gives VS Code access to my Azure subscription that I've just created. Signing in to this took ages, and was eventually tracked down to some kind of conflict with my Skype account that I was logged in to at the same time. The Skype account is linked to my Microsoft account in the same was as Azure is, but VS Code wasn't having it. In the end I logged out of Skype and I was able to login to Azure. Since then I've been able to login to Skype and Azure in any order, so it may have been a glitch with one of the connections at that point in time. If in doubt, sign out of everything!]
  • Visual Studio Codespaces, this allows VS Code to manage and connect to remote dev environments - you don't have to use the browser as a front-end with Codespaces and I can't see any reason you wouldn't be able to do this with Code Builder also, but time will tell.  
You also have to sign in to Codespaces, which you can do via the button on the left of VS Code:


Once you've signed in, choosing to create a new Codespace will take you through a process of creating a new plan - the key questions are the Azure region - make sure to choose one close to you, and the default instance type - I went for Linux.

Putting it Together

Once the accounts and extensions are in place, VS Code gains a new icon at the bottom left:



Clicking this opens the Codespaces menu: 


Choosing 'Create New Codespace' generates a new remote environment, after capturing some information from me. First, what type of environment I want - I've only used the Default settings and had no problems to date.


I then have a choice of creating an empty Codespace, or populating it with the contents of a repository - I'm using my Curated repo (currently private, but the contents of the repo aren't really important for this post - if you are curious what it is, it's my toolbox):


Then an easy question to answer - the Codespace name.


Once all the questions are answered, the extension gets to work on a new Codespace:


As well as the progress bar, a new panel opens on the right of VS Code to show the steps being completed (assuming all goes well - I've had one or two failures, but most of the time it's a breeze);


Once the setup is complete, clicking the Connect button opens the remote workspace through my local VS Code installation, which looks pretty much the same as a local workspace, bar a larger green element on the bottom left of the window:


and the fact that any changes I make here are not reflected in my local workspace unless I round trip them to the version control system. 

This is all well and good, and if I don't have a particularly powerful machine it's a nice way to be able to work on multiple projects at the same time and push a lot of the computing requirement to the cloud. The more interesting environment, however, is the browser.

Codespaces in the Browser

To access my Codespace from a browser, I have to login to the Visual Studio Codespaces web site at : https://online.visualstudio.com/login, using my Microsoft account that I attached my Azure subscription to. As an aside, I only seem to be able to login via a regular Chrome window, not an incognito one. I've no idea whether that is intentional, but it's been consistently the case since I started playing with Codespaces.

Once I've logged in I can see all of my current Codespaces, and I have the option to create more directly from the web site:



Clicking on my new CuratedBlog Codespace opens the IDE in the browser:



And as this is a virtual machine, the terminal works in the browser too!


That's all there is to it?

Not quite - while this gives me a Visual Studio Codespace primed with my Salesforce application metadata and running in the cloud, it doesn't know much about Salesforce, so I have to do a little more setup - note you can configure your Codespace environment via a devcontainer.json file in your project directory, so you wouldn't have to do this every time if you were spinning up multiple codespaces. I haven't looked at this in detail.


Once this is done, you now have the tools to interact with Salesforce orgs, but you still need to authenticate against them. This is the slightly long-winded part as, while it is technically possible to run a browser in a docker container, it doesn't look straightforward, so you'll likely end up using the JWT flow. This isn't particularly difficult, as long as you are familiar with the command line and open SSH commands, but it is a little long-winded as anyone who has set up a CI machine will testify, with the connected apps, keys and certs that you have to set up.

A similar problem presents itself once you have authenticated against your dev hub and created a scratch org - clicking the icon to open the org can't open a browser, so doesn't really do anything. The upside here is that it will typically tell you the URL that it is attempting to open which includes a session id, so you can copy/paste that into your local browser to access the scratch org, even though you haven't got any oauth for this org set up. The URL will appear in the Output tab:


This is where I expect Salesforce to add the Code Builder value - generating a Codespace that has the extensions and CLI pre-configured and already authorised against the org, as well as some other items that I haven't stumbled across yet.

Wrapping Up

While Codespaces require a bit of effort, until Code Builder is more widely available they are a really good way to be able to develop from any device that has a supported browser and internet connection.  In fact I was able to use Chrome on my iPhone XS to push changes to a scratch org. The UI was terrible so I wouldn't recommend it, but as an intellectual exercise it was fun.

Related Posts




Saturday, 30 May 2020

Documentor Plugin - Triggers (and Bootstrap)



Introduction

In previous posts about documenting my org with a Salesforce CLI plugin the focus has been on extracting and processing the object metadata. Now that area has been well and truly covered, it's time to move on. The next target in my sights is triggers - not for any particular reason, just that the real projects that I use this on have triggers that I want to show in my docs!

Trigger Metadata

Trigger metadata is split across two files - the .trigger file, which contains the actual Apex trigger code, and the .trigger-meta.xml file, which contains the supporting information - whether it is active, it's API version. The .trigger file itself contains the object that it is acting on and the action(s) that will cause it to fire:

trigger Book_ai on Book__c (before insert) {

}
Luckily the syntax is rigid - the object comes after the on keyword and the actions appear in brackets after that, so just using standard JavaScript string methods I can locate the boundaries and pull out the information that I want, but not just to display it.

Checking for Duplicates

As we all know, when there is more than one trigger for an object and action, the order they will fire in is not guaranteed. If you are a gambling aficionado then you may find this enjoyable, but most of us prefer a dull, predictable life when it comes to our business automation.

Once I've processed the trigger bodies and extracted the object names and actions, one of the really nice side-effects is that I can easily figure out if there are any duplicates and I can call these out on the triggers page (I've updated the sample metadata to add duplicate triggers - they don't actually do anything so wouldn't really be an issue anyway!) :

Bootstrap

Those that have been following this series of posts may notice that the styling looks a little better now than in earlier versions - this is because I've moved over to the Bootstrap library, pulled in via the CDN so I don't need to worry about including it in my plug-in. 

I've also updated the main page to use cards with badges to make it clear if there are any problems with the processed metadata :



Which I think we can all agree looks a lot better. You can see the latest version of the HTML generated from the sample metadata at https://bbdoc-example.herokuapp.com/index.html

I've also moved some duplicated markup out to an EJS include, which is as easy as I hoped it would be. The markup goes into a file with a .ejs suffix like any other - here's the footer from my pages:

<div class="pt-3"></div>
<footer class="footer">
    <div class="container">
        <div class="row justify-content-between">
            <div class="col-6">
                <p class="text-left"><small class="text-muted">Generated <%=footer.generatedDate%></small>
                </p>
            </div>
            <div class="col-6">
                <p class="text-right"><small class="text-muted">Bob Buzzard's Org Documentor
                        v<%=footer.version%></small></p>
            </div>
        </div>
    </div>
</footer>
And I can include this just by specifying the path:
<%- include ('../common/footer') %>
Note that I've put the footer in a common subdirectory that is a sibling of where the template is stored.
Note also that I've used the <%- tag which says to include the raw html output from the include. Finally, note that I don't pass anything to the footer - it has access to exactly the same properties as the page that includes it. 

As usual, I've tested the plug-in against the sample metadata repo on MacOS and Windows 10. You can find the latest version (3.3.1) on NPMJS at the link shown below.

Related Posts



 

Saturday, 23 May 2020

Going GUI over the Salesforce CLI Part 3

Introduction


In part 1 of this series I introduced my Salesforce CLI GUI with some basic commands.
In part 2 I covered some of the Electron specifics and added a couple of commands around listing and retrieving log files.
In this instalment I'll look at how a command is constructed - the configuration of the parameters and how these are specified by the user.

Command Configuration


As mentioned in part 1, the commands are configured via the app/shared/commands.js file. This defines the groupings and the commands that belong in those groupings, which are used to generate the tabs and tiles that allow the user to execute commands. For example, the debug commands added in the last update have the following configuration (detail reduced otherwise this post will be huge!) :

{
    name : 'debug',
    label: 'Debugging',
    commands : [
        {
            name: 'listlogs',
            label: 'List Logs',
            icon: 'file',
              ///////// detail removed //////////
        },
        {
            name: 'getlog',
            label: 'Get Log File',
            icon: 'file',
              ///////// detail removed //////////       
        }
    ]
}

which equates to a tab labelled Debugging and two command tiles - List Logs and Get Log File :


clicking on a command tile opens up a new window to allow the command to be defined, and here is where the rest of the configuration detail comes into play :


the screenshot above is from the List Logs command, which has the following configuration (in it's entirety) :

{
    name: 'listlogs',
    label: 'List Logs',
    icon: 'file',
    startMessage: 'Retrieving log file list',
    completeMessage: 'Log file list retrieved',
    command: 'sfdx',
    subcommand: 'force:apex:log:list',
    instructions: 'Choose the org that you wish to extract the log file details for and click the \'List\' button',
    executelabel: 'List',
    refreshConfig: false,
    refreshOrgs: false,
    json: true,
    type: 'brand',
    resultprocessor: 'loglist',
    polling: {
        supported: false
    },
    overview : 'Lists all the available log files for a specific org.',
    params : [
        {
            name : 'username',
            label: 'Username',
            type: 'org',
            default: false,
            variant: 'all',
            flag: '-u'
        }
    ]        
}

the attributes are used in the GUI as follows:

  • name - unique name for the command, used internally to generate ids and as a map key
  • label - the user friendly label displayed in the tile
  • icon - the SLDS icon displayed at the top left of the command page
  • startMessage - the message displayed in the log modal when the command starts
  • completeMessage - the message displayed in the log modal when the command successfully completes
  • command - the system command to be run to execute the command - all of the examples thus far use sfdx 
  • subcommand - the subcommand of the executable
  • instructions - the text displayed in the instructions panel below the header. This is specific to defining the command rather than providing help about the underlying sfdx command
  • executeLabel - the user friendly label on the button that executes the command
  • refreshConfig - should the cached configuration (default user, default dev hub user) be refreshed after running this command - this is set to true if the command changes configuration
  • refreshOrgs - should the cached organisations be updated after running this command - set to true if the command adds an org (create scratch org, login to org) or removes one (delete scratch org, logout of org)
  • json - does the command support JSON output
  • type - the type of styling to use on the command tile
  • resultProcessor - for commands that produce specific output that must be post-processed before displaying to the user, this defines the type of processor
  • polling - is there a mechanism for polling the status of the command while it is running
  • overview - text displayed in the overview panel of the help page for this command
  • params - the parameters supported by the command.

Parameters


In this case there is only one parameter, but it's a really important one - the username that will define which org to retrieve the list of logs from. This was a really important feature for me - I tend to work in multiple orgs on a daily (hourly!) basis, so I didn't want to have to keep running commands to switch between them. This parameter allows me to choose from my cached list of orgs to connect to, and has the following attributes:

  • name - unique name for the parameter
  • label  - the label for the input field that will capture the choice
  • type - the type of the parameter - in this case the username for the org connection
  • variant - which variant of orgs should be shown :
    •  hub - dev hubs only
    • scratch - scratch orgs only
    • all - all orgs
  • default - should the default username or devhubusername be selected by default
  • flag - the flag that will be passed to the sfdx command with the selected value

Constructing the Parameter Input


As I work across a large number of orgs, and typically a number of personas within those orgs, the username input is implemented as a datalist - a dropdown that I can also type in to reduce the available options - here's what happens if I limit to my logins :


as the page is constructed, the parameter is converted to an input by generating the datalist entry and then adding the scratch/hub org options as required:

const datalist=document.createElement('datalist');
datalist.id=param.name+'-list';

for (let org of orgs.nonScratchOrgs) {
    if ( (('hub'!=param.variant) || (org.isDevHub)) && ('scratch' != param.variant) ) {
        addOrgOption(datalist, org, false);
    }
}

if ('hub' != param.variant) {
    for (let org of orgs.scratchOrgs) {
         addOrgOption(datalist, org, true);
    }                    
}

formEles.contEle.appendChild(datalist);

The code that generates the option pulls the appropriate information from the cached org details to generate a label that will be useful:

let label;
if (org.alias) {
    label=org.alias + ' (' + org.username + ')';
}
else {
    label=org.username;
}

if (scratch) {
    label += ' [SCRATCH]';
}

and this feels like a reasonable place to end this post - in the next part I'll show how the command gets executed and the results processed.

Related Posts






Sunday, 26 April 2020

Documenting from the metadata source with a Salesforce CLI Plug-In - Part 4


Introduction


In Part 1 of this series, I explained how to generate a plugin and clone the example command.
In Part 2 I covered finding and loading the package.json manifest and my custom configuration file.
In Part 3, I explained how to load and process source format metadata files.

In this episode I'll show how to enrich the field information by pulling in information from elsewhere in the metadata - specifically global or standard value sets. When I'm using a global value set for picklist values, all I have in my field XML metadata is a reference to the set:

<valueset>
    <restricted>true</restricted>
    <valuesetname>Genre</valuesetname>
</valueset>

and the actual values live in the (in this case) Genre.globalValueSet-meta.xml file in the globalValueSets folder.

When I create the HTML report for the object metadata, I want to pull these values in, otherwise whoever is viewing the report has to go into the Salesforce org to figure out just which Genres are available.

Figuring the Value Set Type


Fields using standard value sets don't actually specify the value set name, instead it has to be derived from the field using the information in the metadata API documentation. I try to find a standard value set matching the field and if none exists then I fall back to loading the named global value set. The actual loading is the same, and should be familiar to anyone who read part 2 - I use fast-xml-parser to load the XML and parse - however, as multiple fields can refer to the same value set, I cache the parsed metadata :

let getGlobalValueSet = (dir, name) => {
    let valueSet=globalValueSetByName[name];
    if (null==valueSet) {
        globalValueSetByName[name] = valueSet = 
             parseXMLToJS(join(dir, 'globalValueSets', name + 
                                    '.globalValueSet-meta.xml'));
    }

    return valueSet;
}

I then assign this to a new property on the field named gvs. As I'm processing the metadata in an offline mode, if I can't find the global value set, the property will be null.

Adding the Information to the Field


When processing the field, I expand the values from the value set by iterating them and extracting the names - note that if the gvs property is not populated then I output a message that it isn't in version control:


var vsName=field.valueSet.valueSetName;
result+='<b>Global Value Set (' + vsName +')</b><br/>';
if (field.gvs) {
   field.gvs.GlobalValueSet.customValue.
      forEach(item => result+='&nbsp;&nbsp;' + item.fullName + '<br/>');
}
else {
   result+='Not version controlled';
}

This is a relatively simple piece of enrichment, just pulling some information from another XML metadata file and inserting that in the row for the field. However, it doesn't have to be metadata that is included - it can be additional information, a diagram, anything that can be written out to HTML - quite powerful once you realise that.


One More Thing


The version of the plugin is now at 2.0.3. From the user perspective nothing should change, but the mechanism for generating the report is completely different, which I'll write more about in the next exciting episode.


One More One More Thing



You can view the output for the example repo of Salesforce metadata on Heroku at : https://bbdoc-example.herokuapp.com/index.html

Related Posts




 



Saturday, 18 April 2020

Documenting from the metadata source with a Salesforce CLI Plug-In - Part 3


Introduction

In Part 1 of this series, I explained how to generate a plugin and clone the example command. In Part 2 I covered finding and loading the package.json manifest and my custom configuration file

In this instalment, I'll show how to load and process source format metadata files - a key requirement when documenting from the metadata source, I'm sure you'll agree.

Walking Directories

In order to process the metadata files, I have to be able to find and iterate them. As I'm processing object files, I also need to process the fields subdirectory that contains the metadata for the custom fields that I'm including in my document. I'll use a number of standard functions provided by Node in order to achieve this.

I know the top level folder name for the source format data, as it is the parameter of the -s (--source-dir) switch passed to my plugin command. To simplify the sample code, I'm pretending I know that the custom objects are in the objects subfolder, which isn't a bad guess, but in the actual code this information is pulled from the configuration file. To generate the combined full pathname, I use the standard path.join function, as this figures out which operating system I'm running on and uses the correct separator:

import { join } from 'path';
...
let objSourceDir=join(sourceDir, 'objects');
I then check that this path exists and is indeed a directory - this time while I use the standard fs.lstatSync to check the path is a directory, I use a homegrown function to check that the path exists:

import { lstatSync} from 'fs';
import { fileExists } from '../../shared/files';
 ...
if ( (fileExists(objSourceDir)) && 
     (lstatSync(objSourceDir).isDirectory()) ) {

Once I know the path is there and is a directory, I can read the contents, again using a standard function - fs.readdirSync - and iterate them :

let entries=readdirSync(objSourceDir);
import { appendFileSync, mkdirSync, readdirSync } from 'fs';
  ...
for (let idx=0, len=entries.length; idx<len; idx++) {
    let entry=entries[idx];
    let entryPath=join(objSourceDir, entries[idx]);
}

Reading Metadata

The Salesforce metadata is stored as XML format files, which presents a slight challenge when working in JavaScript. I can use the DOM parser, but I find the code that I end up writing looks quite ugly, with lots of chained functions. My preferred solution is to parse the XML into a JavaScript object using a third party tool - there are a few of these around, but my current favourite is fast-xml-parser because

  • it's fast (clue is in the name)
  • it's synchronous - this isn't a huge deal when writing a CLI plugin, as the commands are async and can thus await asynchronous functions, but when I'm working with the command line I tend towards synchronous
  • it's popular - 140k downloads this week
  • it's MIT licensed
It's also incredibly easy to use. After installing by running npm install --save, I just need to import the parse function, load the XML format file into a string and parse it!

import { parse} from 'fast-xml-parser';

let objFileBody=readFileSync(
join(entryPath, entry + '.object-meta.xml'), 
                  'utf-8');
let customObject=parse(objFileBody);

My customObject variable now contains a JavaScript object representation of the XML file, so I can access the XML elements as properties:

let label=customObject.CustomObject.label;
let description==customObject.CustomObject.description;

Which allows me to extract the various properties that I am interested in for my document.

In the next exciting episode I'll show how to enrich the objects by pulling in additional information from other files.

Related Posts


Friday, 10 April 2020

Documenting from the metadata source with a Salesforce CLI Plug-In - Part 2

Introduction


In Part 1 of this series, I explained how to generate a plugin and clone the example command. In this instalment I'll look at customising the cloned command to locate and load the package.json manifest, and to load a configuration file that defines how my objects should be divided up for documentation purposes. All loading, all the time.

Customising the Command

Renaming

The first thing to do around customising the command is to rename it - as I cloned the sample org command, although it is externally known as bbdoc:doc due to the folder structure, in the source code it is still Org:

export default class Org extends SfdxCommand {

so I change the class name to Doc.

Flags

The next thing is to change the flags supported by the command to those that I need for documentation purposes. The flags for the org command are defined as follows:

protected static flagsConfig = {
  // flag with a value (-n, --name=VALUE)
  name: flags.string({char: 'n', description: messages.getMessage('nameFlagDescription')}),
  force: flags.boolean({char: 'f', description: messages.getMessage('forceFlagDescription')})
};

This shows up that there is slightly more to this than changing the flag names - notice the description properties aren't hardcoded strings. Instead they are retrieved by a message object. Here's the setup for the messages object:


// Initialize Messages with the current plugin directory
Messages.importMessagesDirectory(__dirname);

// Load the specific messages for this file. Messages from @salesforce/command, @salesforce/core,
// or any library that is using the messages framework can also be loaded this way.
const messages = Messages.loadMessages('sample', 'org');


The final line shows that the messages are loaded based on the command name - org.  In the generated plugin, messages are stored in the messages folder and the command specific messages are stored in a file named <command>.json. So for the org command, we have the following file structure:



and org.json has the following contents:

{
  "commandDescription": "print a greeting and your org IDs",
  "nameFlagDescription": "name to print",
  "forceFlagDescription": "example boolean flag",
  "errorNoOrgResults": "No results found for the org '%s'."
}

so as part of defining my flags, I also need to define their descriptions and the description of the command itself as follows:

  • config       - configuration file
  • report-dir  - directory to store the generated report
  • source-dir - source directory containing the org metadata

so I create a doc.json file:

{
  "commandDescription": "generate documentation for an org",
  "configFlagDescription": "configuration file",
  "reportDirFlagDescription": "directory to store the generated report",
  "sourceDirFlagDescription": "source directory containing the org metadata"
}

I initialise the messages with this new file:

const messages = Messages.loadMessages('sample', 'doc');

and update the flags definition in the source code to :

protected static flagsConfig = {
  // flag with a value (-n, --name=VALUE)
  "config": flags.string({char: 'c', description: messages.getMessage('configFlagDescription')}),
  "report-dir": flags.string({char: 'r', required: true, description: messages.getMessage('reportDirFlagDescription')}),
  "source-dir": flags.string({char: 's', required: true, description: messages.getMessage('sourceDirFlagDescription')})
};

Reading the package.json Manifest

The version number of my plugin is stored in the package.json manifest file and I want to include this in the generated HTML, so it's time to locate and read that file. I know the package.json is at the root of my plugin directory, but I don't know where the plugin has been installed. Node has a standard File System module, fs, that has been extended by Salesforce to add a handy utility function - traverseForFile. This starts from the folder the command lives in and makes its way up the folder hierarchy until it finds he requested file, or hits the root directory for the disk.  I execute this to find the location of the package.json file, which is in the plugin root:

const pluginRoot=await fs.traverseForFile(__dirname, 'package.json');

Having found the directory, I then read the package.json file. Note that I'm using the join function from the standard Path module - this allows me to build a path without worrying about the operating system or path separator. Note also that as reading a file is an asynchronous operation, I use the await keyword to stop the plugin until the read is complete.

// get the version
const packageJSON=await fs.readFile(join(pluginRoot, 'package.json'), 'utf-8');

Having read the JSON from the file, I parse it into a JavaScript object using the standard JSON.parse function and extract the version property:

const pkg=JSON.parse(packageJSON);
let version=pkg.version;

Loading the Configuration

I find I'm using configuration files more and more often with plugins. While I could add my configuration to sfdx-project.json, and I do if it is only a few items, when I start getting into lengthy configuration it feels like I'm polluting that file with things that don't belong there.

For this plugin and the example Salesforce metadata, I have the following configuration in bbdoc-config.json:

{
    "objects": {
        "name": "objects",
        "description": "Custom Objects", 
        "subdirectory": "objects",
        "suffix":"object",
        "groups": {
            "events": {
                "name":"events", 
                "title":"Event Objects",
                "description": "Objects associated with events",
                "objects": "Book_Signing__c, Author_Reading__c"
            }
            ,
            "other": {
                "name":"other", 
                "title":"Uncategorised Objects",
                "description": "Objects that do not fall into any other category"
            }
        }
    }
}

The objects property contains the configuration for generating the HTML document from the object metadata. It's in its own property as I intend to add more capability in the future. There's some information about the metadata, where it can be found, the suffix for the metadata files, and then a couple of groupings for the objects - the event specific objects and the rest.

The configuration file is optional - if one isn't provided there will be a single object grouping of uncategorised, so I check if he flag was supplied before trying to load any file.

// load the config, using the default if nothing provided via flags
let config;

if (!this.flags.config) {
  this.ux.log('Using default configuration');
  config=defaultConfig;
}
else {
  config=await fs.readJson(this.flags.config);
}

This time I'm using another Salesforce extension to the fs module - readJSON - which reads and parses the file into an object in one go - I should probably refactor the loading of the package.json file to use this!

In the next instalment, I'll show how to load and process the metadata source files. As before, if you can't wait, you can see the plug-in source on Github or install it yourself from npm.


RELATED POSTS