Marc Doucet: Software Automation Analyst
February 15, 2024
During a previous assignment, I was tasked with maintaining a solution for generating massive amounts of data, entirely built with Postman. Typically used for testing API calls, Postman can also be used to write pre-request scripts and tests in JavaScript. In our case, the Postman solution contained complex programming: a thousand lines of code spread across several interdependent modules.
Description of the Initial Solution
To facilitate understanding of the solution, let's limit the description to this: a tool used in development environments to generate clients and insurance quotes
This generation requires CSV files, one for client generation and one for quote generation. These files contain the necessary input values: first name, last name, address, etc. Each line in a data file represents an input iteration (a client or a quote). Some of these values can be generated randomly. The use of CSV files to parameterize tests is a basic feature in Postman's Collection Runner module.
The complexity of programming this solution lies in orchestrating API calls based on the contents of the data files. Postman lacks code editing or debugging tools like those found in an integrated development environment (IDE). You can't navigate the code by ctrl-clicking on dependencies. This limitation, combined with the complexity of the programming, made maintaining or enhancing the source code very difficult.
To simplify maintenance, the team had undertaken to copy all the Postman coded components into a monolithic file, deployed with Node.js. This solution was only used to develop new features and fix bugs. For each addition or correction in Node.js, it was necessary to manually copy the modified parts back into Postman.
Description of the First Refactoring
Since the Postman solution was divided not only into collections but also into modules, the monolithic aspect of the Node.js solution made comparing and synchronizing the two solutions quite arduous. We therefore opted for a refactoring of the Node.js code, dividing it into modules to try to mirror the structure established in Postman as much as possible. For each request, we wrote three modules: the pre-request, the request, and the post-request representing respectively the Pre-request Script, Body, and Tests tabs in Postman. We placed these requests in directories reflecting the Postman collections. We also developed a module emulating Postman's functionalities: variable management, random values, identifier management, request objects, and response objects, etc.
Thus, it became much easier to navigate and test the code.
However, we still had to deal with certain constraints:
· Maintaining code in two places, although facilitated by the common structure, remains vulnerable to oversight errors.
· The code, being unprotected, could be modified inadvertently.
· The execution report is only available in the log text.
· The user has to search through the logs to find errors.
· The absence of Git; we use the free standalone version, which complicates versioning. Even if we version the Node.js solution, the Postman solution only has a version in the name we give it.
· The usability of manipulating CSV files is limited, the user opted for a data file in Excel format which he had to convert to CSV to integrate into Postman.
· The Postman Collection Runner module is not easily accessible.
· The user must copy data collected during one step to paste it into a data file used for the second step (i.e., he must retrieve the client number for creating a quote).
Beyond the Postman solution, we now had a fully functional and well-structured Node.js tool. So...
How to make this solution autonomous?
How to abandon the use of Postman and no longer maintain code in two places?
For security and cost reasons, it was not possible to deploy this new solution as a web application. It was also not feasible to provide the source code to users and invite them to deploy the Node.js application locally.
The Electron Environment
During our research, we discovered Electron, an environment that allows creating desktop applications written in JavaScript. Electron uses Chromium and Node.js. It's like deploying a Node.js application and interacting with it in Chrome.
In other words, Electron allows writing a 100% web application and converting it into a standalone application. This application can be deployed on both Mac and Windows or Linux. Not to mention that creating installers for these platforms is very simple.
On the UI side, beyond pure HTML, Electron is compatible with React, Vue, etc.
The environment also interacts with the operating system, which allows, among other things, using local files, configuring application menus, using alerts, and ensuring data and configuration persistence.
Furthermore, Electron is at the core of several widely used applications, including:
Teams
Postman
VSCode
Skype
Slack
Although very robust, constantly maintained, and with a large user community, the environment still has some drawbacks:
The generated application can be large since it encapsulates Node.js and Chromium.
Chromium is known for its high memory consumption, and Electron unfortunately inherits this problem.
It is still cumbersome to deal with the memory issues even if optimization strategies are well documented in this regard.
In March 2023, Microsoft unveiled its rewrite of the Teams application, abandoning Electron in favor of their in-house solution, WebView2. More information can be found here.
In April 2023, Electron announced the establishment of a team specifically focused on performance. This announcement, perhaps circumstantial, nonetheless testifies to the seriousness of the Electron development team.
Description of the Autonomous Solution
To convert our Node.js solution, the base code did not need any modification, except for the "Main" which contains most of the configuration and the addition of files specific to Electron. Finding answers to our questions was greatly simplified thanks to the vast user community. Within two days, we developed a functional proof of concept with a user-friendly HTML interface.
This new solution surpasses others due to the following advantages:
A minimalist graphical user interface.
Uploading a single data file in Excel format without needing conversion.
Execution presented as a table containing more information, including error messages (if any) in the execution line itself.
Report generation in CSV, PDF, and Excel formats.
The possibility of having a copy of all request and response payloads as separate files in the report directory.
A single codebase to maintain.
Concrete versioning with version numbers in the About menu.
Simplified distribution, with a single versioned installer.
The Future
No longer constrained by Postman's paradigms, we can refactor the code to streamline its structure and, in many places, speed up executions. This complete detachment from the Postman solution makes the development of new features possible. We are planning to work on:
A graphical user interface allowing 100% random generation based on probability percentages.
A different graphical user interface for each type of operation.
Using React to optimize the UI.
Programmatic orchestration of the data file, containing tabs for each step. This prevents manipulation errors likely to occur when transferring data by copy-pasting between steps.
For our service offering, converting the Postman solution into an Electron application was a logical choice. We gained efficiency in code maintenance, adding new features, and distributing the application. We easily adapted the graphical user interface to meet various user needs, and we are no longer dependent on an application that can change its functionalities at any time. Additionally, if deploying a web application becomes feasible, we would already be fully compatible.
When we discovered Electron, we quickly agreed that we were facing the only serious solution of its kind. Since then, other players made their debut, such as Tauri and React Native Desktop (developed by Microsoft).
Electron meets our expectations and needs. Its constant development, numerous extension modules, and available resources reinforce our conviction that we made the right choice.
Comments