macOS security – Secure Kernel Extension Loading (SKEL)

Some background first

As we get ready to approve kernel extension for our Apple fleet, here is some background on how things evolved.

Kernel extension (KEXT) – MacOS kernel combines the features of a microkernel (load the minimum to run the OS separately from other services) and a monolithic kernel (everything needed for the OS, drivers and filesystem are in the kernel space like BSD). In MacOS, kernel extensions are used by different applications to grant low level access to the hardware. For example, some software, like Antivirus, require access to the computer hard disk and memory to function correctly.

Depending on the skill level of the developers writing these apps, some kernel extensions might be poorly written. There are some apps that don’t even remove the kernel extension when you uninstall them. Others can cause random crashes to your OS or slow down your machine. Apple wants to give the user control over the applications that are installing kernel extension on our Macs and gives the MacAdmins a way to approve them for your fleet.

History

Let’s take a look how the security of Mac OS evolved over time.

OS X 10.8 (Mountain Lion, and earlier, 2012) – Digitally signed kexts are not allowed and wont load.

OS X 10.9 (Mavericks, 2013) – Signed kexts will load but not enforced.

OS X 10.10 (Yosemite, 2014) – Signed kexts enforcement. Your kext will not be loaded if it’s not digitally signed.

OS X 10.11 (El Capitan, 2015) – SIP is now taking care of the code signing verification.

OS X 10.13 (High Sierra, 2017) – Secure Kernel Extension Loading (SKEL) is managed now by System Integrity Protection (SIP).

User Approved Kernel Extension Loading (UAKEL) was introduced to give users control over which kernel extension are allowed to load on their Mac during the software installation.

User Approved MDM (UAMDM) – As an admin, you want to manage certain security-sensitive settings on a Mac to keep security tight. If you use DEP, you are good to go since the device already belongs to your account. The other option is to manually “Approve” additional management privileges from the users computer.

The idea behind this is that some actions have to be approved by the user of the Mac so that a sneaky MacAdmin won’t be able to run items on a user’s personal device without their knowledge. Any user, not only computer administrators, can approve a kernel extension unless restricted by the MDM.

You don’t have to enroll to an MDM to approve kernel extensions manually and can approve them when notified during installation, via the security menu in system preferences.

A few important notes

  • When you are upgrading to macOS High Sierra or later, all third-party KEXTs that were already installed at the time of upgrading are automatically approved and don’t require any user action.
  • Automation or attempting to enroll a device remotely via screen sharing will not result in a “User Approved enrollment”. You have to connect a mouse to the device and click a physical button. You can connect via Apple Remote Desktop app and use a USB mouse to click a button if there is no screen connected to the device.
  • When approving KEXTs manually and if not approved during installation, the approval “Allow” button is only available for 30 minutes after booting, will reappear after a restart in the “security and privacy” -> “general” tab. However, you will no longer see that pop-ups asking you to approve the kernel extension

Back to kernel extensions, once a device is “User Approved” you can use the MDM to whitelist the kernel extension you approve in your organization. More on that below.

Let’s get technical

To identify the KEXT, macOS uses Team Identifier to identify the vendor or Bundle Identifier to identify the specific application installing the KEXT. Here is a community maintained list of known identifiers :

https://docs.google.com/spreadsheets/d/1IWrbE8xiau4rU2mtXYji9vSPWDqb56luh0OhD5XS0AM/edit#gid=1070689416

If you want to whitelist the vendor like “Google” you can use their Team ID “EQHXZ8M8AV”. This means that any future KEXTs by Google are automatically approved. If you prefer to whitelist Google File Stream you can use the Bundle ID “com.google.dfsfuse.filesystems.dfsfuse”. However, you must specify the Team ID in both cases

Location of Kernel Extension

Approved and system KEXTs can be found in:

/System/Library/Extensions
/Library/Extensions

Newly installed 3rd party KEXT are copied to the folder mentioned below and once approved, a copy of the extension is made to the one of the folders above and allowed to load.

/Library/StagedExtensions

Working with KEXTs on your mac via CLI

/usr/libexec/PlistBuddy -c print /Library/Extensions/Dropbox.kext/Contents/Info.plist
  • List approved kernel extensions on the Mac
kextstat | grep -v com.apple
  • Find the location of the kernel extension
kextfind -bundle-id com.getdropbox.dropbox.kext

kextfind -b com.getdropbox.dropbox.kext
  • Unload until next reboot
sudo kextunload /System/Library/Extensions/NAME_OF_THE_KEXT_FILE.kext
  • Load until next reboot
sudo kextload /System/Library/Extensions/NAME_OF_THE_KEXT_FILE.kext
  • Remove kernel extensions (should back up just in case)
sudo rm -r /System/Library/Extensions/NAME_OF_THE_KEXT_FILE.kext
  • Get the Team-ID and Bundle-ID of kernel extensions on your Mac
sudo sqlite3 /var/db/SystemPolicyConfiguration/KextPolicy "SELECT team_id,bundle_id,allowed,developer_name,flags from kext_policy"
  • Get the list of MDM approved kernel extensions
sudo sqlite3 /var/db/SystemPolicyConfiguration/KextPolicy "SELECT team_id,bundle_id,allowed,payload_uuid from kext_policy_mdm"
  • Get the audit history of kernel approvals on your Mac
sudo sqlite3 /var/db/SystemPolicyConfiguration/KextPolicy "SELECT path,team_id,bundle_id,boot_uuid,created_at,last_seen,flags from kext_load_history_v3"

! Any modifications to the Kernel Extension User Consent must be done from the Recovery OS only !

  • Add the Team Identifier into the list allowed to load kernel extensions without user consent (from Recovery OS)
sudo spctl add <team-id>        
  • Remove the Team Identifier (from Recovery OS)
sudo spctl kext-consent remove <team-id>   

The wrong way to deal with it

If you want to remove this security feature from your environment and are ok with the risk, you can turn off the Kernel Extension User Consent from the Recovery OS. Not recommended.

  • Boot to recovery and check status of KEXT approval (bad idea)
sudo spctl kext-consent status
  • Boot to recovery and turn off KEXT approval (bad idea)
sudo spctl kext-consent disable
  • Boot to recovery and turn on KEXT approval (bad idea)
sudo spctl kext-consent enable

Resetting NVRAM will reset those changes. You can set a firmware password to prevent unauthorized changes.

The easy life with an MDM (JAMF in this case)

As mentioned you must enroll the computer via DEP or “Approve” the newly enrolled device manually. The best way is to use the user enrollment web page by going to:

https://your.jamf.server:8443/enroll

Add  a separate JAMF account for each member on the team that can only perform enrollments. This way you’ll have an audit trail of enrollments to your MDM and it simplifies monitoring.  

This will install a certificate (if its self-signed) and an MDM config profile. Once this is done, you can push the config profile with the approved KEXT Team or Bundle ID payload to the device.

The approved kernel extension payload must be pushed via an APNS transaction from the MDM server to a Mac. A profile with this payload will not work if it is installed via script and/or package.

Use a separate Config profile and apply it to the Mac before deploying the software that installs the KEXT. You can to deploy all of the kernel extension to all company computers.

Give the profile a general name like “Approved Third-Party System Extensions” to avoid questions from users.

Finally the JAMF payload itself

Now let’s create and deploy the KEXT approval payload with the MDM. Create a new configuration profile and define the desired scope. Add the Approved Kernel Extension payload with the relevant details. Here is that community google sheet link again:

https://docs.google.com/spreadsheets/d/1IWrbE8xiau4rU2mtXYji9vSPWDqb56luh0OhD5XS0AM/edit#gid=1070689416

Save and ….. Done.





Sources:

APFS- Apple file system

As I was reading the book “macOS Installation” by Armin Briegel, I realized I need to learn more about APFS, here is a summary of what I found.

Why APFS is here

  • HFS is 30 years old, was designed when there were no permission, no hard links, no extended attributes and checksum in mind. Apple added and corrected many times over the years which cause the system to become less efficient and …. kinda crashy. The biggest issue with that is that files on HFS could get corrupted over time and you would lose your precious photos after all those years.
  • Apple needed a file system for its newer devices like Apple watch and AppleTV
  • The increasing use of SSD technology. APFS is optimized for flash and solid-state drives and leverages their high seek and access speeds, much faster than spinning drives.
  • Improving the encryption and making it hog less CPU.

The benefits of APFS

  • Optimise the space of the HDD used and increase free space by using clones for file copying and using deltas.
  • Improve the write speed by removing the need for journaling and switch to “write on copy” and by changing the way file are stored and pointed to on the disk. This is efficient on SSD. When you make a copy of a file no blocks are copied and a pointer is added from the new file to the previous file’s data. If you make changes to the original file, APFS created new records in the file system with the changes that you made. This is possible due to the low access time of SSD vs HDD that take time to spin the drive.
  • 64-bit file system that helps with space savings and performance improvements
  • Snapshots – they are allowed by the “write on copy” that enables you to “copy” a huge chunk of data within seconds
  • End users can store up to nine quintillion files on a single volume, boom! (only 4 billion today))
  • Natively support encryption vs Filevault 2 which was part of the OS and not the filesystem. This allows encrypting parts of the files for different purposes or users.
    You can create a separate encrypted partition for your personal files.
    Also possible to encrypt specific files or metadata.
  • Apple also changed the way to decrypt the APFS partition. The Preboot volume has the data needed to boot to the FileVault preboot login screen and to decrypt the volume (Using the T2 chip to verify integrity, more on that in future posts)

APFS container

  • With APFS an abstraction level called “containers” was added. Volumes now reside in containers and those volumes can “share the free space” meaning they can change without repartitioning. Yey!


The downside of APFS

  • Not compatible with HFS+ or NTFS . MacOS 10.12 and before will not be able to access it.
  • Should not be used on HDDs due to “write on copy” and the “fragmentation” of copied files. HDDs have low seak speed and will be slowed down by this feature
  • Does not support direct hard link, will be converted to soft links during the conversion (During an upgrade to a new OS)

What can we do with it

  • Take a snapshot of your Mac APFS container during testing and revert to its original state instead of reinstalling. Read this article by the awesome Emily to get some info.

Here are some of the commands:

tmutil     - make snapshots on apfs command (old time machine control)

Ctrl+Option+Command+T    - Open terminal during Setup Assistant and Mac installation.

tmutil snapshot - take the snapshot

tmutil listlocalsnapshots / - list the snapshots

tmutil deletelocalsnapshots 2018-12-12-083439 - delete specified snapshot

tmutil thinlocalsnapshots / 1000000000 1 - remove older snapshots based on priority
  • Take a snapshot before a major upgrade or add this to you scripts or Jamf self service before upgrading the OS.
  • Snapshots are kept locally for up to 24 hours and also erased based upon free disk space. You could create a script or use launchctl (or crontab) to take a snapshot every X amount of time without having the Time Machine drive connected for local

Resources used:

Changing wallpaper on new users computer

I’m learning to love Jamf every day and this time it’s the simple task of setting the wallpaper for newly installed computers

.

There is a simple way to setup a Wallpaper with a configuration profile. However, this will force the wallpaper and people will not be able to replace it later. I did not want that functionality.

I found a nice method that I’ll describe here that ended up working for me. One thing to note is that I do not have DEP and the policy runs from the users account that I manually create on the Mac.

The first part is straightforward. I packaged the wallpaper PNG file to be deployed to the users’ document folder with the Jamf composer. I used FEU (Fill existing user home directories) to make sure it goes to the users home folder and does not create one with my username in the path. Then I exported to a dmg file as .pkg files will not work with FEU.

Next, I used this amazing tool called Desktopper on Github. Packaged it and deploy the .pkg in the same policy. With Desktopper you can point it at the PNG file I copied to the users’ Documents folder and set it on the some or all displays as the wallpaper.

I could use the “Files and Processes” section of the policy but then the desktoppr will run as the root user and will not give me the result I want.

The solution I ended up using is called Outset and can be found on Github and…. it is awesome. Outset is a script which automatically processes packages, profiles, and scripts during the boot sequence, user logins, or on demand.

What I need to do is put create a script with the correct permissions, that will run the desktoppr and set the PNG file. That script went to the “login-once” folder of Outset. This way the script will apply the wallpaper after the first login for every user on this Mac. Added it to the scripts in Jamf. This was the last piece of the puzzle.

#!/bin/bash
## Create a script in the outset Run-Once folder
cat <<EOF>> /usr/local/outset/login-once/SetWallpaper.sh
#!/bin/bash
/usr/local/bin/desktoppr main ~/Documents/FBwallpaper.png
EOF

## Give the script execute permission
chmod +x /usr/local/outset/login-once/SetWallpaper.sh

As scope, I set up a new smart group that includes all of our user machines that were enrolled in the last day. I do not want this to replace the wallpaper of my CEOs favorite cat or something. So that’s it, guys. Please let me know if you have a better way to get the job done or any questions.

Working with Plist files

In this article we will go over .plist files, defaults command and some software to work with plist files.

Background info

All of the Macs preferences are stored in plist files (property list file). Some settings are not accessible from the GUI of the MacOS or its applications preferences menus. You might also want to configure an application before a user launched it on a new Mac he just got : For example, you might want to set the default home page for a browser, adjust the screen to lock after three and a half minutes or set the VPN configuration.

Every time you make a change to a setting via any menu, those changes are reflected in the relevant plist file by updating an entry and setting a value on that entry in the relevant plist file. It looks like an XML file, but if you try to open it, you will not be able to read it with a text editor as it is a binary file. You need a program to interact with those files. This is where the defaults command comes in.

We can take a look at the many .plist files located at:

~/Library/Preferences/

/Library/Preferences/

Preview plist files

MacOS has a nice feature that allows preview to display the content of a plist file that is readable. Once you select the file you want to view, hit space, and the content will be displayed.

Quick edit plist files

When you are using the defaults command you need to specify if you want to read a plist or update it.

defaults read com.apple.finder

View only the value of AppleShowAllFiles. The result can be TRUE, FALSE, YES or NO.

defaults read com.apple.finder.plist AppleShowAllFiles

Let’s change the value so that we can see all files and verify it was changed.

defaults write com.apple.finder.plist AppleShowAllFiles YES

Converting plist files to text

You might need to convert the plist file into something more human readable. Use the plutil command to convert plist to txt. I use this when I need to take the content of a plist file and paste its content to JAMF to deploy it. Make a copy of the file you will be converting and…

plutil -convert xml1 com.apple.finder.plist

You can reverse the conversion using:

plutil -convert binary1 com.my.app.plist

Applying what we learned + some useful apps.

You would like to change the homepage for all your Macs in the Chrome browser to your company portal.

  • The first thing to decide on is this a user or a computer policy and how will you be deploying it
  • Now you need to know where is the file that you need to edit. This can be done by Googling the file to update and then using the defaults command.
  • You can also look at the file system changes when you are making that change on a virtual MacOS. I love an app called http://fsmonitor.com/. It gives you this nice tree look of the live updates to the OS.

  • Another super great tool is Pref Setter. It will show you a list of all the plist found on the Mac and gives you an option to update from inside the app. I would not recommend using it to update, it’s quite old, but it is great to get a general view.

  • PlistEdit Pro is my tool of choice for editing. It gives a great visualization and simplifies editing. There is also a more affordable alternative here.

  • There is also a tool that is provided by Apple called plistbuddy.. I have not used it yet but heard it mentioned several times on the MacAdmins podcast.

Adding default settings to packages

Let’s say that that you are building a package with JAMF composer and you need to add some default setting for that app. Once you add the app and use a post-install script to run the defaults command with the values you want. Make sure to build a .pkg file though if you want it to work correctly.

Well, that’s it for now guys. While doing some research for this article, I found a book by Armin Briegel (https://scriptingosx.com) . His book Property Lists, Preferences and Profiles for Apple Administrators is only available with a US Apple ID. Perhaps future books will be as he mentioned on his blog post.

Fully loaded Mac – in the past

We covered the features Apple is giving us to manage our macs in the previous post. However, some of those features are not yet available in Israel. They will show up here eventually, possible with Apple Business Manager release in the next few months.

In the meantime let discuss how we used to deploy new Mac, what worked for us so that later we can talk about the direction we are moving in. Let me know in the comments if you have any suggestions or questions.

Administrative tips

First, we purchase our Macs through a big company that will have significant interest to be approved to be a DEP reseller once it is available in Israel. This way we will probably be able to ensure that previously purchased equipment will be added to our account.

Once the Macs arrive, I do not open them right away since Apple’s one year warranty starts at the moment you activate them (connect to the internet for the first time). I do have a few spare Macs ready to be given just in case someone forgot to let me know about a new person that has already started working … yesterday.

I consider it best practice to have the same configuration across all the fleet per job role and the same color of devices for all. All RND folks get the same machine, and all support get the same weaker configuration. It is simpler to manage, troubleshoot and does not cause people to complain about their hardware.

As soon as I find out a new person is starting, I will fire up that Mac and ….

Deployment workflow

Here is what I used to do so far, later I’ll go into what we do now. So the first thing I did was go into netboot mode on the Mac and put a primary image on the laptop. Even if there was a working version of MacOS, I would erase it. Never know who touched the Macs while in transit and also want to have a consistent setup for all of our devices.

We used Deploystudio and had an image for every computer type. This does add the task of keeping the images up to date, but it is relatively simple as you create a new image after a new deployment and an upgrade of the OS and drop it back on the deployment server. The more diversified your Mac fleet is, the more images you will need to maintain.

Part of the Deploystudio workflow is to copy files to the Mac, install packaged and other security features once the image is finished deploying. We used that to install some packages that will create the admin user and kickstart the Munki binary installation. Munki is a great open source project that is widely used and provides you with an internal “App Store” on the Mac with company approved software. We would also copy a few files over to avoid running around with a USB drive and copying the default wallpaper.

Munki opened so many options for us! We used to rely on Apple Remote Desktop to install things remotely, but this creates many checklists and was not prone to errors. With Munki you can create different lists of packages and configurations that it will apply consistently. We had separate lists for different departments too. We could also update the software for the whole company or a specific department. Running scripts is also an option so you if you want to use the defaults command to set settings or bash CLI that will work correctly.

If you are using an MDM, one more step you can do is enroll your Macs using a quick add package. I prefer doing that at the Deploystudio level (not Munki), so I can be sure that once a Mac finished without errors, it is now enrolled.

To speed things you should use a wired cable an create a separate VLAN only for the deployment purposes. It is possible to use an external drive and that might speed things up however this is a single point of failure. Also if it takes 5 min or 30, it does not make that much of a difference as long as you don’t need to be there pushing buttons and clicking things.

So now you have a Mac, with the latest OS, enrolled to your MDM, connected to Munki, pulling all the apps and setting. We put a printed page with the first-day instruction inside the laptop and back in the box it goes, waiting for the new owner to open it up.

Monitoring

You get some indication from Deploystudio that things ran successfully, but that was pretty much it so far. We needed a solution to monitor the status of the Macs and let us know if something is going wrong. After testing several options, we decided to use Watchman Monitoring. It was everything we needed for a reasonable price.

This service is fantastic at what it does:

  • Asset tracking
  • Hardware and software information
  • Warranty details
  • Notification on errors

You can get notifications for:

  • Kernel Panic
  • Computer name or OS changes.
  • OS downloads
  • Full HD

We could customize the alerts to our liking and open a ticket to our helpdesk software for those situations when we want to follow up.

That is how it used to work, but then Apple came in and had to change things…

Management framework by Apple for the MacAdmins of today

The last few weeks were intense. I was away from my family, my day to day life, habits and work because I was doing the JAMF 200 and 300 courses and certification one after the other in a matter of two weeks.

We have been integrating JAMF for our Mac and iOS devices after working with open source programs for a while. I just got tired of the constant maintenance that deprived me of the ability to move forward on the many projects in the pipeline.

So what is JAMF?

JAMF is an Apple product management solution (not developed by Apple though). It will allow you as a Mac Admin to do what you want and asked for with your Mac fleet. While other proprietary solutions focus on integrating the MacOS and iOS in the windows environment, the main advantage of JAMF is that it’s solely focused on the Apple ecosystem and on all of its aspects. Working closely with Apple they are usually the first ones to implement new features. The fact that only one OS needs to be maintained probably simplifies the system overall complexity of their system and the possibility of bugs.

It’s a collection of tools that can simplify the day to day action There is a large community of professionals on the JAMF forums who are really helpful with any question and their support has been helpful so far, even without the 24/7 support plan. Of course, this comes at a price of about 60$ per computer per year, s no-brainer if you consider the number of technicians it can replace if implemented correctly, more on that later.

I will not go all into what JAMF does cause you can get all that info on Google but I would like to share with you how we are using it today.

Let us start with some background.

Apple is making a big step in helping Mac Admins manage their devices. And they do it the way Apple does everything, their own way. Apple is providing tools for installing software and settings remotely while maintaining security and its focus on user privacy.

This does come at a cost for the Mac Admin who got used to doing some things their own way and does not want to invest more time in tinkering with something that is already working (smart guy right 🙂

However, this will not work for long. Apple introduced some software and hardware changes over the years that will require us to adjust.

  • SIP (System integrity protection) that no longer gives you access to some of the system folders.
  • The latest Macs come with the T2 that completely blocks the option of full drive imaging.
  • User approved kernel extensions. This feature makes sure that the person who owns the computer is aware of the systems that have access to the system and hardware code.

Many were concerned with the way things were going, but Apple did listen to feedback (so far) and provided new methods to manage devices. Let’s welcome some abbreviations :

MDM – Mobile device management – A solution to manage devices. Apple does not provide one or plan to do so. They did, however, provide protocols, systems, and guidelines how an Apple MDM should work and set up the Apple-controlled backbone so that the controlling power would stay with Apple. This way they can guarantee the quality and security of their product and service

APNS – Apple Push Notification Services – This is the magic sauce that Apple is using to send commands between your Mac and the MDM to get things rolling and to perform some tasks.

DEP – Device enrollment program. This is an amazing feature. If only it was available here in Israel. What it does, is kickstarts enrollment of a Mac into your MDM of your choice. The moment you purchase any Apple device from an approved Apple reseller, its serial is automatically added to your account and with a few clicks in JAMF, you can assign it to a profile, install software and other useful things. You don’t even have to do it yourself. Ask the reseller to ship the laptop to the users home and the laptop will start “setting itself up” once it is connected to the internet for the first time. (With the latest MacOS, each computer must connect to the internet to perform activation). You (with Apple’s help) can control any aspect of the installation

VPP – Volume Purchase Program – Similar to volume license, but better. This way your company can purchase software and assign it to some users. The software is owned by you and the license is transferable. The user does not need to have an Apple ID to install any software

Apple Business Manager – This should unite the VPP and DEP into one. Supposedly it will launch in Israel and bring DEP too (fingers crossed)

GSX – Also not available in Israel yet. This allows you to see the insurance information of your device and simplify the ordering of spare parts.

JAMF – Is using all of the Apple services mentioned above to provide us, Mac admins, with an awesome experience managing Macs.

Now that we have that covered you know the Apple side lets move forwards. MacOS has several options to manage and control Macs:

  • Use the defaults command to add or modify settings, very versatile.
  • Use MacOS CLI commands that perform different actions with the OS.
  • Apple configuration profile, it’s a file that determines and enforces what your device can or cannot do. Awesome for things like making sure encryption is turned on or that the computer locks after a few mins of inactivity.

Apple really does not want us to tinker with the OS too much so that it will work as it intended. This can be clearly visible by the limiting the power of the MacAdmin and the root user of the Mac.

What they ended up doing is provide is a simpler way for us to get all that simply by toggling the features on and off in the MDM of our choice. Not all features are available but the most important ones are there.

In the next post, we will go into the workflow we use to provision MacOS devices from the moment we receive the equipment until the user of that new Mac is happily working with his new toy.