Optimization month!

This month's updates to the project focused on extensive optimizations, security enhancements, and code cleanup, with around 100 improvements made in these areas. GenDoc and widgets enhancements, Caprover integration with easier docker installation support, Survey enhancements and another set of bug fixes complete yet another incredible month of evolution.

We have detected and fixed an error in the Record Access Control functionality this month. The evaluation of conditions to apply the control or not was evaluated in the context of the administration user instead of the currently logged-in user. In the majority of conditions this doesn't make any difference. Most of the time Record Access Control conditions are set on field values present in the record. In the particular use case where we found the error, we were calculating the conditions based on the role of the current user. In general, any record access control conditions that used the getCurrentUserField and similar workflow expressions were being evaluated incorrectly and applying the wrong permissions. If you have any record access control permissions configured make sure they are not using those functions, and, if so, review that the results are what you expect. This is a breaking change for a certain type of RAC rules.

• ask for record emails popup when sending merged template by email. This works only on those modules that already support that feature. Note that adding that popup to any module should be as easy as adding an email field to the module. Killing yet another PDFMaker functionality!
• ask for confirmation fix
• check document/template integrity on save for better error detection
• supporting environment setting for GenDoc_Convert_URL and GenDoc_Convert_URL_UnoServer as per the new Global variable enhancement commented below.
• get the OnlyOffice global variable with the correct module context
• fix error getting Business Map ID from foreach condition
• denormalized documents query support
• direct support for Company image without relation
• eliminate spaces and special characters from merged template calculated name to avoid conversion errors
• standardize uitype 10 fields in the direct merge screen
• send execution and workflow context information when doing an OnlyOffice edit and save. This change is very powerful. The use case is when you are editing a merged template in the OnlyOffice integration and you need to execute a workflow when the document is saved. Since the "save" event of the document being edited will be triggered you can launch a workflow on that event, but, that workflow would launch on ALL the save events of the document record. We need a way to know or detect that the save event is happening from the OnlyOffice edit. That is where this change comes into play. We have enhanced the application to permit sending workflow context when saving a record from inside the application. This can be done in any save event so we can add context to any integration. In fact, before this change, we used to do this by adding a checkbox field to the module which was flipped when saving from the integration. In this case, we added these context variables:

oosave => true,
gendocsave => true if editing started from GenDoc, false otherwise,
oorecord => CRMID of the record that initiated the merge if coming from GenDoc, 0 otherwise,
oomodule => module name of the record that initiated the merge if coming from GenDoc, empty string otherwise,

This permits us to set a condition like this to trigger the workflow only when saving from OnlyOffice:

The cronWatcher scheduled task is in charge of checking, every few minutes if any of the other scheduled tasks has entered a failing state. When it detects that state it sends a warning to the administration user.

Normally, what we do is reset the failed task and the problem is not repeated. This is due to edge cases that are difficult to reproduce and fix. So we decided to add that functionality directly into the cron Watcher task. Now, when the cron watcher detects a failed task, it will reset it back to active and increment a counter. In the next check if it is back in a failed state and the counter is bigger than the accepted number of resets then it will start sending warnings. It is like an auto-healing service.

The accepted number of resets is defined by the Global Variable CronTasks_cronWatcher_maxResets

This month sees the definitive deprecation of the obsolete Document Folder picklist. With the change implemented this month, we eliminate all the obsolete code that supported the document folders based on the picklist and we convert that picklist into a standard picklist named Document Group. From now on we will leverage the Document Folders module.

• convert Documents to a normal module, eliminate attachmentfolders and uitype 26

CapRover is an extremely easy-to-use app/database deployment & web server manager for your ... Evolutivo!

We add an automaticInstaller feature along with a set of continuous integration changes to support Caprover setups.

The Survey Path module gets a last set of features to make it implement a many-to-many relation between Surveys and Questions. Survey Path defines the sequence of questions that should be presented to the user when doing the survey and also applies a score on each question.

The survey path module permits us to select questions as if they were a pool of questions that we can repeat in any Survey and also supports defining questions to be skipped depending on the answers given in each question.

This set of changes also establishes a link between Survey answers and Survey Path

• we refactor the search functionality so it can be loaded into any module. Basically, by loading the SearchUI.tpl with some preset Smarty variables we can get the same search functionality we have in the list view into any extension or module
• new CREATEWIDGET business action type, for the create popup widget
• Master Detail support related fields
• complete error and success attributes in all custom action buttons
• support WSID and CBUUID everywhere: with this feature, we can now use the CBUUID identifier in business maps, conditions and anywhere we want
• make __mastergridid field optional
• add transformation map as an Import map parameter
• support adding one value as a string directly with setPicklistValues
• Updater gets a new orderBlocks mass action
• getTablenameByFieldname function
• add two parameters to the Execute Script Business Question functionality: recordid and report_parameters, which permit us to inject context into the script being executed
• support for setting workflow context when we save a record programmatically inside the application: ExecutionContext
• GenerateDocument Workflow task support injecting template IDs to generate through context: GenDoc_TemplateIDs_List
• get My IP: ping.php?myip=1 show your detected IP

• EvolutivoFW Standard Formatting
• Documentation: code and wiki.
  • function headers and help messages
  • correct workflow aggregation examples
  • wiki documentation keeps going
• Refactor and Optimize code
  • SonarQube recommendations, reduce cognitive complexity, change for loops, eliminate nested ternary operations all over the application
  • substitute variables for their value, reduce SQL columns, reduce query_result, simplify debug message
  • change obsolete HTML font
  • eliminate an unsupported connection parameter in PEARDB
  • compact Global Variable calls in javascript to reduce network traffic
  • add a local cache for Actor Module primary key search which is called a lot
  • eliminate obsolete actions from index.php
  • move email and WhatsApp duplicate code into a script (attachmentManager)
  • give the Workflow conditions more horizontal space
  • eliminate AMD/Node structure from Advanced Filter. It is not our environment
  • add table indexes. thanks to the MySQL optimization project we detected a set of tables that require indexes
  • convert EntityCache to a singleton for it to work as intended: as a cache. this is an important optimization to reduce a lot of database access and speed up the workflow system (which is used everywhere)
  • recalculate privileges only for active users and force automatic recalculation of privileges when user is set back to active
• Security fixes and updates
  • update Session to 4.1.0
  • correct .htaccess settings to permit access to some files in Smarty templates when bettersafe script is applied
• PHP constant updates
  • HTML entities mb_convert_encoding deprecated fix
  • bump application support to PHP 8.3
• CI: is now correctly running again
  • adding test php syntax step, syntax error fixing using PHPCS
  • drone docker type unit tests
  • made the repository public
  • make the sendMessageToMMUser script work in the docker type
  • adding the Mattermost step sent to the author on failure
  • the DRONE_FAILED_STEPS is not given to us when using the docker type
  • using secret instead of plain text
• Unit Tests: keeps getting more and more assertions.

This month we implement functionality that permits us to inject global variables from the execution environment of the application. This will be mostly used in docker configurations, but could also come in handy for other types of installations. If the global variable name, preceded by the prefix GV_ is present in the operating system environment it will take precedence over any global variable definition in the database. In fact, the application will not even look for it in the database.

This enhancement is related to the Caprover/docker project and is available only for developers to use. In other words, only developers can decide which variables support environment definitions. This is the list of supported global variables (for the moment):

• Application_Storage_Directory
• Application_Storage_SaveStrategy
• Application_Upload_MaxSize
• Application_Minimum_Cron_Frequency
• Application_UI_Name
• Application_UI_CompanyName
• Application_UI_ShowGITVersion
• Application_UI_ShowGITDate
• Application_UI_Version
• Application_UI_URL
• Application_Unique_Identifier
• Webservice_Enabled
• WEBDAV_Enabled
• Notifications_Enabled
• Zero_Bounce_API_KEY
• GenDoc_Convert_URL
• GenDoc_Convert_URL_UnoServer
• ip_elastic_server
• ip_elastic_indexprefix
• esusername
• espassword
• Apache_Tika_URL
• OnlyOfficeDocumentServer_UR
• Workflow_Maximum_Scheduled

We added one new global variable this month, which is CronTasks_cronWatcher_maxResets, as explained above.

We also eliminate these two:

• Document_Folder_View: while eliminating the old folder view as explained above.
• SOAP_CustomerPortal_Enabled: SOAP support was deprecated many months ago, so we cleaned up this unused variable.

• delete Global Variable records when the Global Variable value is deleted
• Updater was doing an incorrect interation on field names in the order fields mass action
• escape HTML elements in the email popup screen
• eliminate obsolete code for PHP 5
• set language and user for the send 2FA code functionality
• admin user traceability for Business Maps and validations
• AuditTrail: set user ID in the record column on login
• use the correct function name in Colorizer
• use the correct entity module for condition expressions
• do not edit the relation if the relation field is set in CreatePopup
• Detail View
  • use CreatePopup as an action and load it in Detail View
  • recover support for code blocks and inline related lists. Lost when we did the new detail view blocks structure for sorting
  • skip blocks with empty fields
  • check if the related lists array is present and fix a javascript regression error
• ListView
  • prevent a fatal error in SQL and show the correct screen to the end user
  • return error message on a failed query instead of undefined
  • get related modules from the session correctly and optimize the foreach loop
• DMS
  • add CSS property for the name column
  • add preview and edit for XLS file
  • create documents via web service and remove the dependency from ID 0 for the root files
  • remove a wrong relation with the parent module
• Master Grid
  • add view mode in API
  • apply Field Dependency based on the Mode setting from the business map
  • correct function name and render uitype 77
• Documents
  • convert to a normal module eliminating DocumentFolders picklist-specific code
  • denormalization fixes in DocumentPreview, attachment search, GenDoc, and a few other places
  • sync detailview.php code to support create view widget
  • send error file when trying to download an empty file
  • set default values and modify integrity test to be faster and avoid warning message
• Emails
  • add the full subject to hover in the related list
  • show specific error message in the notification panel if given
  • json_decode may return null instead of array
• support Field Dependency for special fields account_id and contact_id
• support FQN in fieldDep_GetField assignment
• increment home field specification length
• avoid duplicate entries in vtiger_inventorysubproductrel and vtiger_inventoryshippingrel when creating an Invoice from Sales Order Recurring Invoicing
• Login: move the send code link and open on lost focus to make it easier to use when autofill is active
• MailParser: encode special chars into UTF-8
• silence Mail Scanner output when the CRON quiet parameter is set
• return empty JSON array instead of warnings when no template IDs are given in the Message Templates module
• fix Popup Filter using background data in detail view and format date fields to ISO format
• QueryGenerator: relate users through assigned_user if not relating on another field and support all open user fields
• skip validating empty conditions on Reports Advanced Search
• retrieve the MAX of role ID correctly
• format parent fields in the tree view widget
• User traceability: condition error to find admin ID user
• prevent creating users with the same username
• return correctly formatted values in the userNameOrGroupNameExists function
• Workflow
  • eliminate obsolete Events reference
  • allow edit for multi-picklist fields to MassiveUpdate workflow task
  • cast to integer for MySQL save
  • save Execution and Workflow task context wherever possible
  • eliminate obsolete code from UI change
  • Email and WhatsApp Tasks: denormalized Documents queries, document selection and presentation when MailManager is deactivated
• Translations

Finally some changes in the performance indicators! Restore my faith

June	July

Thanks for reading.