Posted on April 20th, 2023 by Agathocles Prodromou, CNO, 3CX Mandiant identifies the source of internal network compromise While Mandiant’s investigation is still ongoing, we now have a clear overall understanding of the attack. Following our previous update, we would like to share some additional technical details to support our customers and the community. We have also published additional indicators of compromise that organizations can leverage for their network defenses.
Initial Intrusion Vector Mandiant identified the source of our internal network compromise began in 2022 when an employee installed the Trading Technologies X_TRADER software on the employee’s personal computer. Although the X_TRADER installation software was downloaded from the Trading Technologies website, it contained VEILEDSIGNAL malware, which enabled the threat actor (identified as UNC4736) to initially compromise and maintain persistence on the employee’s personal computer.
The X_TRADER installer (X_TRADER_r7.17.90p608.exe) was digitally signed by a valid code signing certificate with the subject of “Trading Technologies International, Inc”. It was hosted on hxxps://download.tradingtechnologies[.]com. While the X_TRADER software was reportedly retired in 2020 by Trading Technologies, the software was still available for download on the Trading Technologies website in 2022. The code signing certificate used to digitally sign the malicious software was set to expire in October 2022.
Lateral Movement Following the initial compromise of the employee’s personal computer using VEILEDSIGNAL malware, Mandiant assesses the threat actor stole the employee's 3CX corporate credentials from his system. VEILEDSIGNAL is a fully-featured malware that provided the threat actor with administrator-level access and persistence to the compromised system. The earliest evidence of compromise uncovered within the 3CX corporate environment occurred through the VPN using the employee's corporate credentials two days after the employee's personal computer was compromised.
Additionally, Mandiant identified the use of the Fast Reverse Proxy tool (https://github.com/fatedier/frp) which the threat actor used to move laterally within the 3CX environment. The tool was named MsMpEng.exe and located in the C:\Windows\System32 directory.
CI/CD Build Environment Compromise Mandiant’s investigation was able to reconstruct the threat actor’s steps through our environment as they harvested credentials and moved laterally. Eventually, the threat actor was able to compromise both the Windows and macOS build environments. On the Windows build environment, the attacker deployed the TAXHAUL launcher and COLDCAT downloader which persisted by performing DLL hijacking for the IKEEXT service and ran with LocalSystem privileges. The macOS build server was compromised using a POOLRAT backdoor using LaunchDaemons as a persistence mechanism.
Attribution Based on the Mandiant investigation into the 3CX intrusion and supply chain attack thus far, they attribute the activity to a threat actor cluster named UNC4736. Mandiant assesses with high confidence that UNC4736 has a North Korean nexus.
Indicators of Compromise X_TRADER_r7.17.90p608.exe SHA256: fbc50755913de619fb830fb95882e9703dbfda67dbd0f75bc17eadc9eda61370 SHA1: ced671856bbaef2f1878a2469fb44e9be8c20055 MD5: ef4ab22e565684424b4142b1294f1f4d
Going Forward Our priority throughout this incident has been transparency around what we know as well as the actions we’ve taken.
As we wind down our incident investigation, 3CX has taken this opportunity to continue to strengthen our policies, practices, and technology to further protect against future attacks. With that, we’re announcing a 7 Step Security Action Plan. In this plan, we’re committing to actionable steps to harden our defenses. You can read in more detail here.
Initial Results from Mandiant Incident Response Following the appointment of Mandiant as our security incident response team, forensic analysis on our network and product is in progress. In a nutshell, the interim assessment concluded:
Attribution Based on the Mandiant investigation into the 3CX intrusion and supply chain attack thus far, they attribute the activity to a cluster named UNC4736. Mandiant assesses with high confidence that UNC4736 has a North Korean nexus.
Windows-based Malware Mandiant determined that the attacker infected targeted 3CX systems with TAXHAUL (AKA “TxRLoader”) malware. When executed on Windows systems, TAXHAUL decrypts and executes shellcode located in a file named .TxR.0.regtrans-ms located in the directory C:\Windows\System32\config\TxR\. The attacker likely chose this file name and location to attempt to blend into standard Windows installations. The malware uses the Windows CryptUnprotectData API to decrypt the shellcode with a cryptographic key that is unique to each compromised host, which means the data can only be decrypted on the infected system. The attacker likely made this design decision to increase the cost and effort of successful analysis by security researchers and incident responders.
In this case, after decrypting and loading the shellcode contained within the file .TxR.0.regtrans-ms was a complex downloader which Mandiant named COLDCAT. It is worth noting, however, this malware differs from GOPURAM referenced in Kaspersky’s report.
The following YARA rule can be used to hunt for TAXHAUL (TxRLoader):
rule TAXHAUL { meta: author = "Mandiant" created = "04/03/2023" modified = "04/03/2023" version = "1.0" strings: $p00_0 = {410f45fe4c8d3d[4]eb??4533f64c8d3d[4]eb??4533f64c8d3d[4]eb} $p00_1 = {4d3926488b01400f94c6ff90[4]41b9[4]eb??8bde4885c074} condition: uint16(0) == 0x5A4D and any of them }
Please note that in a similar way to any YARA rule, this should be properly assessed within a test environment first before usage in production. This also comes with no guarantees regarding false positive rates, as well as coverage for this entire malware family and eventual variants.
MacOS-based Malware Mandiant also identified a MacOS backdoor, currently named SIMPLESEA, located at /Library/Graphics/Quartz (MD5: d9d19abffc2c7dac11a16745f4aea44f). Mandiant is still analysing SIMPLESEA to determine if it overlaps with another known malware family.
The backdoor written in C communicates via HTTP. Supported backdoor commands include shell command execution, file transfer, file execution, file management, and configuration updating. It can also be tasked to test the connectivity of a provided IP and port number.
The backdoor checks for the existence of its configuration file at /private/etc/apdl.cf. If it does not exist, it creates it with hard-coded values. The config file is single-byte XOR encoded with the key 0x5e. C2 comms are sent over HTTP requests. A bot id is generated randomly seeded with the PID of the malware upon initial execution. The id is sent with C2 communications. A brief host survey report is included in beacon requests. Message contents are encrypted with the A5 stream cipher according to the function names in the binary.
Persistence On Windows, the attacker used DLL side-loading to achieve persistence for TAXHAUL malware. DLL side-loading triggered infected systems to execute the attacker's malware within the context of legitimate Microsoft Windows binaries, reducing the likelihood of malware detection. The persistence mechanism also ensures the attacker malware is loaded at system start-up, enabling the attacker to retain remote access to the infected system over the internet.
The malware was named C:\Windows\system32\wlbsctrl.dll to mimic the legitimate Windows binary of the same name. The DLL was loaded by the legitimate Windows service IKEEXT through the legitimate Windows binary svchost.exe.
Command and Control Mandiant identified that malware within the 3CX environment made use of the following command and control infrastructure:
Update 7 Windows Electron App build number 18.12.424 has been checked by our advisors Mandiant who found no evidence of compromise. Due to certificate changes, we had to update the naming convention. We re-opened the build to address this. Build 18.12.425 has been created and released. Below is our most up to date information.
Still Preferred Option: Install 3CX as a Native Web App (PWA)
Installing 3CX as a native app (PWA) is still our preferred and recommended option. To summarise:
There’s no need to be logged into 3CX with a tab open When a call or chat comes in you’ll be notified with a PUSH notification Depending upon your browser, this is what you can expect: Microsoft Edge, your browser will be auto started Google Chrome, your browser will NOT be auto started if closed Safe and practical use it together with iOS and Android apps or deskphone to control calls on your smartphone or deskphone Setting the App to Auto Start
To ensure zero admin, along with automatic updates and secure running within the browser, we recommend auto starting your browser on system startup. Read the instructions on how to set the app to auto start.
How to Download Windows & Mac Electron
Ensure Your Server Has the Latest Update Installed
Important! First, ensure your server has the latest update installed. Details about this and the steps, if any, that you need to take are shown below.
Customers on 3CX Hosted / StartUP - No action needed
3CX Hosted and StartUP users do not need to update their servers as we will be updating them automatically today. Servers will be restarted and the new Electron App MSI/DMG will be installed on the server. We still recommend that you use the PWA App.
Posted Apr 06, 2023 - 08:34 PDT
Update
We are continuing to monitor for any further issues.
Posted on April 1st, 2023 by Pierre Jourdan, CISO, 3CX
The Desktop App can be uninstalled as explained below. On some Windows machines where antivirus software already deleted some of the files the uninstaller may fail.
On Windows:
Start Type “Control Panel”, Enter Select “Programs and Features” Find 3CX Desktop App, select and press “Uninstall”. On Mac:
Go to “Applications” Tap on “3CX Desktop APP” Right click then “Move to Bin” Ensure that it isn’t also present on Desktop otherwise delete it from there as well. Empty the Bin Mass / Network Uninstall of Electron App
Partners on our forums have kindly contributed Powershell scripts that allow companies to mass uninstall the electron app from their Network. We have merged them into one that will attempt to uninstall and forcibly delete any remaining files and entries associated with the Desktop App. We hereby thank the original authors of the scripts that were merged. This powershell script hasn't been thoroughly tested yet from our end, we recommend testing it on one machine first before executing it on your customers infrastructure. This must be run on client machines not the server.
Important: There are many scripts being suggested on the internet. Please be careful with any script or executable found on the internet, do not blindly trust them as they may be harmful.
Continuing to monitor, latest recommendations are to use the PWA application until newly signed Windows and MacOS Desktop Applications are made available, see link below for more details on the current secure options
Posted on March 31st, 2023 by Nick Galea, CEO, CTO & Founder, 3CX
Unfortunately Google has invalidated our software security certificate. This means the MSI DesktopApp files that we released yesterday afternoon can no longer be downloaded via Google Chrome (nor can the originally infected MSI files). Furthermore several AV vendors are blocking any software signed with the old security certificate.
We are now going to make new MSI installers with a new certificate. This will take at least 8 hours, as we have decided to build a completely new build server. The following MSI installers have been blocked:
SBC for Windows Windows Desktop APP Call Flow Designer
We repeat that we recommend using our PWA app. The PWA app is completely web based and does 95% of what the Electron App does. The advantage is that it does not require any installation or updating and chrome web security is applied automatically.
The MAC desktop application will not be rebuilt for the time being as we focus on the Windows app as well as the actual security breach.
Posted Mar 31, 2023 - 08:31 PDT
Update
We are continuing to monitor for any further issues.
Posted Mar 31, 2023 - 08:20 PDT
Update
We are continuing to monitor for any further issues.
Posted Mar 30, 2023 - 16:38 PDT
Update
We are continuing to monitor for any further issues.
Posted Mar 30, 2023 - 14:53 PDT
Update
3CX DesktopApp Security Alert - Mandiant Appointed to Investigate
Posted on March 30th, 2023 by Nick Galea, CEO, CTO & Founder, 3CX
Early this morning we informed our partners and customers that our electron windows app shipped in Update 7, version numbers 18.12.407 & 18.12.416, included a severe security issue. We since learned that Electron Mac App version numbers 18.11.1213, 18.12.402, 18.12.407 & 18.12.416 have also been affected. Fortunately, anti-virus vendors flagged the executable 3CXDesktopApp.exe and blocked it.
3CX Appoints Leading Incident & Forensics Company Mandiant In response to this, 3CX has appointed Mandiant a renowned American cybersecurity firm and subsidiary of Google - and the market leader in threat intelligence. With their help we will be able to review this incident in full. Whilst their investigation is underway, we ask you to follow the instructions below immediately.
Ensure Your Server Has the Latest Update Installed Customers on 3CX Hosted / StartUP - No action needed 3CX Hosted and StartUP users do not need to update their servers as we will be updating them over the night automatically. Servers will be restarted and the new Electron App MSI/DMG will be installed on the server. We recommend that you DO NOT install or deploy the Electron App. This update is only to ensure that the trojan has been removed from the 3CX Server where Desktop Apps are stored and in case any users decide to deploy the app anyway. During the restart there might be disruption for a few minutes while we restart your server.
Self-Hosted and On-Premise - Install update For Self-Hosted and On-Premise follow these steps:
Launch Management Console Go to Updates Download Mac Desktop App - 18.12.422 Download Windows Desktop App - 18.12.422 On the Clients / Desktops Uninstall the Electron App Follow these steps to uninstall the Electron App for Mac or Windows
For Windows:
Start Type “Control Panel”, Enter Select “Programs and Features” Find 3CX Desktop App, select and press “Uninstall”. On Mac:
Go to “Applications” Tap on “3CX Desktop APP” Right click then “Move to Bin” Ensure that it isn’t also present on Desktop otherwise delete it from there as well. Empty the Bin Use PWA instead of the Electron APP - Here's how Install the Web Client as an app (PWA)
Login to the Web Client You have two options: Click on the OS icon below the user avatar. A new dialog will open, select “Web App (PWA)” and then hit the “Install” button. OR click on the “Install button” (A screen with an arrow) located in the address bar and confirm. See the icon circled red in the screenshot. To set the app to auto start: On Google Chrome: Open your Chrome browser and type ‘chrome://apps’ into the address bar. Right click on “3CX” and enable “Start app when you sign in”. On Microsoft Edge: On Edge, select to Auto-start in the dialog that appears after installation. PWA only works on Google Chrome and Microsoft Edge - not on Safari or Firefox
You can read more in the Web Client user manual.
Avoid Using the Electron App Unless Absolutely Essential In a day or two from now, we will have another Electron App rebuilt from the ground up with a new signed certificate. This is expected to be completely secure. We strongly recommend that you avoid using the Electron App unless there is absolutely no alternative. The Electron App update that we are releasing today is considered to be secure but there is no guarantee given that we only had 24 hours to make the necessary adjustments.
More Information to Come - Transparency Assured We are still working to decipher the full extent of the attack and we promise full transparency as soon as we are clear on everything. We don’t want to jump the gun and make wrong assumptions. Please follow our forum and blog as well as our LinkedIn, Twitter, Facebook and Instagram pages as we’ll continue to update our customers and partners regularly.
Our Continued and Very Sincere Apologies We continue to offer our very sincere apologies to all our partners and customers worldwide. The entire 3CX team continues to work around the clock.
Posted Mar 30, 2023 - 10:41 PDT
Update
Update 30 March 2023 - 8:48 AM
3CX have released Desktop App version 18.12.422 to patch the security issue, please update the desktop app to this version as soon as possible
Posted Mar 30, 2023 - 08:49 PDT
Monitoring
Dear Client,
3CX Security Notice - 30 March 2023
What is the issue, and what does it affect?
We regret to inform our customers that the 3CX Electron Windows App shipped in Update 7, with version numbers 18.12.407 & 18.12.416, includes a security issue. Anti Virus vendors have flagged the executable 3CXDesktopApp.exe and in many cases uninstalled it. Electron Mac App version numbers 18.11.1213, 18.12.402, 18.12.407 & 18.12.416 are also affected.
The issue appears to be one of the bundled libraries that were compiled into the Windows Electron App via GIT. 3CX are still researching the matter to be able to provide a more in depth response later today.
Here’s some information on what has been done so far.
What has been done to mitigate the impact by 3CX?
Domains Have Been Taken Down
The domains contacted by this compromised library have already been reported, with the majority taken down overnight. A github repository which listed them has also been shut down, effectively rendering it harmless.
It is worth mentioning - that this appears to have been a targeted attack from an Advanced Persistent Threat, perhaps even state sponsored, that ran a complex supply chain attack and picked who would be downloading the next stages of their malware. The vast majority of systems, although they had the files dormant, were in fact never infected.
New Windows App in Progress
Currently, 3CX are working on a new Windows App that does not have the issue. They have also decided to issue a new certificate for this app. This will delay things by at least 24 hours so please bear with them whilst this happens.
What can you do until the new desktop App is ready for release?
Use the PWA App Instead!
We strongly suggest that you use the 3CX PWA app instead. The PWA app is completely web based and does 95% of what the electron app does. The advantage is that it does not require any installation or updating and chrome web security is applied automatically.
The reason 3CX has two apps is that when the 3CX Electron App was released, the PWA technology was not available yet. Now it's mature and working really well. More information on how to install it here. One deficiency at this time is the lack of the BLF (Busy Lamp Field) Option in the PWA dial pad.