• Osquery Daemon And Shell
• Osquery
• Osquery-java

This blog post is part twenty-one of the 'Hunting with Splunk: The Basics' series. This series is now legally allowed to drink! (Which is relevent when dealing with James Brodsky as a contributing author.) osquery has quickly become one of the most used resources for gathering data on *nix and MacOSs. However, along with its ability for real-time information pulls, it's also a phenomonal tool for logging. With that in mind, James goes over some interesting things he has found and how to do them! – Ryan Kovar

Author's Note: Ryan Kovar is a taskmaster. He doesn’t understand that my current job at Splunk is to “manage” a bunch of security people—not to actively do hands-on experimentation and write up explanations of security-useful data sources. However, he asked me to come out of semi-retirement for this blog on osquery. If you see him, tell him he owes me a chilled bottle of Loire Valley rosé and that I want my Smiths mixtapes back.

Manufacturer: osquery project; Version: 4.1.2; Website: Description. Osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. Data Model Coverage driver. You need to enable JavaScript to run this app. You need to enable JavaScript to run this app.

Osquery: Osquery comes ready built with a wide variety of log forwarders and this is one of the few places I would actually recommend combining functionality. Log shipping is a relatively mundane task, so the real answer here is use whatever works best for your environment. Osquery uses basic SQL commands to leverage a relational data-model to describe a device. Osquery is an operating system instrumentation framework for Linux that exposes this operating system as a high-performance relational database so that SQL queries can explore the operating system data.

As part of recent work within the Security Specialist team at Splunk, we’ve been taking a closer look at sources of endpoint data that we're seeing across our customer base. It isn’t easy—every time I open my Twitter feed, someone's going off about the latest open-source this and next-gen that. Because guess what? It turns out you can throw all the endpoint++, machine learning, AI-based detection and prevention technology you want at your systems, but unless you’ve got 100% coverage, you still have risk. I base this on findings from the SANS 2018 Survey on Endpoint Protection. The overwhelming majority of people don’t fully implement what they buy.

(This is exactly what happened when I bought that P90x DVD last year so that I could look just like Doug Merritt.) Wineskin winery mac catalina.

But I digress. One of the more prevalent endpoint data sources out there is osquery; let's go deeper.

Osquery is a fully open-sourced, scalable endpoint visibility project contributed to and sponsored by Facebook. Since it's multi-platform (Linux, Windows, and macOS) and tested at scale, it's increasingly likely to be found on corporate endpoints. Other companies such as Palantir and Kolide actively extend osquery, and the popularity of the project is such that QueryCon—an inaugural conference on all things osquery—was held in San Francisco this summer. (The videos from those sessions are excellent, high definition, sometimes irreverent, and can be found at that link as well.)

We’ve been seeing a gradual uptake in Splunk customers looking to analyze data at scale from osquery, especially on non-Windows endpoints. (We have yet to see a customer running osquery in production on Windows, and the one customer seriously considering it was such a huge Microsoft fan that they then gave us a VIP tour of their collection of Zunes.)

Customers looking for a solution for macOS endpoint monitoring, for example, often end up running osquery—as well as those looking to satisfy file integrity monitoring (FIM) requirements. The most common way to onboard osquery data is the time-tested Splunk Universal Forwarder, which can be configured to send local osquery output in real time directly into your Splunk indexers.

Why Should You Care?

Osquery is a fantastic tool for hunting through granular endpoint data. For example, in the Boss of the SOC (BOTS) version 2.0 dataset, we collected osquery data from two macOS endpoints. We used osquery for FIM capabilities against typical download destinations and configured customized process monitoring as well. This picked up a particularly poorly-written (but effective) piece of ransomware that we executed as part of the data generation for the competition; as you can see below, the full command line auditing was captured in Splunk. By using the data returned from “columns.cmdline,” competitors could actually decrypt files encrypted by the ransomware, peel back another layer of our warped sense of humor, and score lots of points! In fact, columns.cmdline (and really anything related to process monitoring with osquery) is some of the most useful security data harvested from Linux and macOS systems. (We’re still partial to Sysmon and 4688 Security events from Windows platforms).

Osquery is often configured with “query packs,” which are simply collections of osquery queries gathered together under a common purpose; much like the revolutionaries in Les Miserables. When you configure osquery to send data from standard query packs into Splunk, the “name” field is populated with the name of the pack that generated the data. Here, we’ve marked the data as coming from the two OS variants in our data generation environment (Windows and Linux.):

You’ll notice above that many of the query packs are focused on operational aspects of your systems. That’s right, osquery is just as useful for your IT Operations staff as it is for your security analysts. IT staff can quickly find performance issues and query their entire osquery fleet to check on configuration settings, installed apps, and hardware details, and then they can get back to sleep. But returning to security, you’ll notice many packs that include query categories like “Windows Hardening” and “Vulnerability Management” and “Incident Response” in the name.

What are examples of just some of the questions we can answer easily in Splunk with those packs?

• What Linux systems have had recent changes to /etc/hosts, and what are those changes?
• Which systems are suddenly listening on new ports (or no longer listening on existing ports), and what are those ports?
• Which users are currently logged in?
• What modifications have been made to Linux crontabs?
• Why does Arizona taunt me with their crazy timezone bullshit?
• Which Linux systems are now pointing to new APT repos?
• Which Windows systems are no longer configured to use UAC?
• Which Windows systems have potentially had a stickykeys backdoor installed?
• Which Linux kernel modules are currently in use?

Querying Splunk for this info couldn’t be simpler. When osquery finds a change of interest (based on the delta from the last time a check was run), it sends the data in JSON format directly into Splunk. Then, you simply query the pack name, hosts, and timeframe to see the output. Here’s an example of looking for that stickykey backdoor check from the standard Windows Attacks detection pack (this attack is when Windows binaries normally used for system accessibility are swapped out with binaries that can be used for evil, which is exactly what I did for this example):

On October 1st, we invite you to come and play BOTS 3.0 at .conf18 and experience osquery data from Windows and Linux endpoints yourself! You’ll encounter the data from all of the packs listed above. You'll drink beer and eat expensive conference canapés. We’ll also be covering, in related BOTS sessions, some of the latest and greatest Splunk integration points with osquery, including a recently-updated app that turns your Splunk search head into a full-fledged osquery query server.

See you in Orlando!

From https://osquery.io/:

Fleet¶

Security Onion includes Kolide Fleet to manage your osquery deployment. For more information, please see the Fleet section.

Agents - Deployment¶

To deploy an osquery agent to an endpoint, go to the Security Onion Console (SOC) Downloads page and download the proper osquery agent for the operating system of that endpoint. Use so-allow to allow the osquery agent to connect to port 8090 on the manager. Then install the osquery agent and it should check into the manager and start showing up in Fleet.

Osquery will attempt to connect to the Manager via the Manager’s IP or Hostname - whichever was selected during the Manager setup. If the hostname is used, the endpoints need to be able to resolve that hostname to the Manager’s IP. See this value by running the following command on the Manager: sudosalt-callpillar.getglobal:url_base. If this value ever changes, the osquery packages under Downloads will need to be regenerated.

All the packages (except for the macOS PKG) are customized for the specific Grid they were downloaded from, and include all the necessary configuration to connect to that Grid. The macOS package is a stock Launcher package, and will require additional configuration once it has been deployed.

For macOS deployments, install the package and then configure the following:

• Update /etc/so-launcher/secret with the Fleet enroll secret. This can be found by running the following on the Manager:

• Update /etc/so-launcher/launcher.flags - change the hostname to your Manager hostname, and change the port from 443 to 8090
• Update /etc/so-launcher/roots.pem with the contents from the following file (on your Manager): /etc/ssl/certs/intca.crt

At this point, osquery should connect up to Fleet within a couple minutes - if not, try to manually restart the osquery agent on the macOS endpoint:

Agents - Updating¶

Security Onion uses Kolide Launcher as a management wrapper around Osquery. This allows for a simpler configuration as well as auto-updates of Launcher and Osquery from the Kolide TUF service. Launcher will check every hour to see if an update is available and, if so, will download and install it. This is the default configuration, but can be changed within the osquery Flags file.

In an airgap environment where the endpoints do not have Internet access, updated Osquery packages can be downloaded from the Security Onion Console and used to update the endpoints. Osquery packages are periodically updated on the Manager as new versions of Osquery are released.

Agents - Troubleshooting¶

Osquery Daemon And Shell

Agent logs on Windows endpoints can be found under the Application channel in the Windows Eventlog - source is Launcher.

Agents - Regenerating Install Packages¶

To regenerate packages, run the following on the Manager (it will take up to 5 minutes to rebuild the packages):

Hunt or Kibana¶

1password password reset. All osquery logs can be found by using the following query:

Kibana Dashboard: Host Data –> Modules/Osquery¶

This dashboard gives an overview of the osquery logs in the system. As long as the default osquery configuration is used, this dashboard should work out of the box regardless of how you schedule or name your queries and packs.

Shipping Windows Eventlogs¶

Windows Eventlogs from the local Windows system can be shipped with osquery to Security Onion. Current parsing support extends to core Windows Eventlog channels ( Security , Application , System ) as well as Sysmon under the default channel location. These logs will show up in Security Onion as event.dataset:windows_eventlog or event.dataset:sysmon.

• Confirm that you can successfully live query the logs: SELECT*FROMwindows_eventslimit10;
• Save a new query: Query -> Manage Queries -> Create New Query SELECT*FROMwindows_events; -> Save
• Add the new query to a query pack that targets a Windows host - how often it should run depends on log volume on the local host; start off with 180 seconds, differential logging: Packs -> Manage Packs -> Select + Edit Pack (Modify Targets for Windows only if needed, Modify Logging options as needed)
• Save pack + Enable pack, if needed.

Please refer to the osquery documentation for further information on osquery Evented tables: https://osquery.readthedocs.io/en/stable/development/pubsub-framework/#the-pub-sub-evented-data-framework-of-osquery

Community ID¶

We sponsored the development of Community ID support for osquery:

Osquery

More Information¶

See also

Osquery-java

For more information about osquery, please see https://osquery.io/.