Configure Event Queries
- The event query area lets you search for specific events using built-in queries or custom queries.
- You create your own conditions for each search. For example, for events during a specific period in specific customer networks.
- The results table shows all events which match the query conditions. The table also lets you run a look-up on external IPs involved in the event.
Once created, an event query can also be used for:
- Custom dashboards which show the query results as charts. See 'Configure Custom Dashboard' for more details.
- Correlation rules which identify harmful events/incidents and assign them to customer admins for attention. See Manage Rules for more details.
To open the 'Event Query' interface:
- Click the menu button at top-right > 'Investigation' > 'Event Query':
- The left-hand panel shows predefined and custom queries for the selected customer.
- The main panel shows the parameters of the selected query and its output.
- The 'New Query' tab contains a query builder. This allows you to create granular, customer queries for the selected customer. Any queries you create will be added to 'Custom queries'.
- Click 'Search' to run the query.
Event Query Interface - Table of controls |
|
---|---|
|
Select the customer on whose behalf you want to run the query. |
Search for a particular query. Enter the name of the query fully or partially and click on the search icon or press 'Enter'. The queries matching the entered text will be listed. To view the full list of queries again, clear the search field and press 'Enter'. | |
Expand or collapse the list of queries. To collapse, click the first button and to expand it, click the second button. Click the refresh button at the end to instantly update the query list. |
|
|
Add a new 'Queries' folder to the left side panel. |
Edit the name of a 'Queries' folder. |
|
Add new event query under a selected query folder. |
|
Delete selected query folders or event queries. |
|
Import saved queries. |
|
Export queries. | |
|
Add conditions for a query. The options available from the drop-down are:
|
Expand or collapse the upper pane to view the complete list of conditions in the query. |
|
Configure the 'Results' table for the query displayed in the upper pane. |
|
|
Save a newly created or edited event query. |
|
Save the query under a different filename or to a different folder. |
Save and move your query to another folder. |
|
Configure alerts and email notifications based on the quantity of events detected by the query within a specified period. See explanation under 'Configure duration based alerts' for more details. | |
Refresh displayed data. | |
Run a query search for time-period. You can set the start end dates to search for events matching the conditions defined in the query and click the 'Search' button in the 'Advanced Search' dialog that appears on clicking this button to view the list of events. Please note the event query created for searching events in the specific period in the past using this option, cannot be saved. |
|
Choose the time period from which events are fetched. Periods range from 1 hour to 7 days. |
|
|
Run a search operation. |
The interface allows administrators to:
- Manage a query folder
- Add and Manage event queries
- Configure results table for a query
- Configure duration based alerts
- Run event queries
- View Results Table
- Perform IP Lookup of External IP Addresses from results using IPVOID
- Perform Lookup of IP Address/Domains from results using Virus Total
- Add Field values to Live Lists from results
- View Aggregated results
- Update and Refine Queries from results
- Export queries
- Import queries
- Export query results to CSV file
Query folders contain collections of event queries. Every new query must be placed in a query folder.
- Choose the customer from the 'Customers' drop-down at the top of the left panel.
Predefined queries added for a customer are displayed in a tree structure in the 'Queries' pane.
- Choose the parent folder to create a new sub-folder and click the button. The Folder Name dialog will appear.
- Enter the name for the folder and click the 'Add' button
The folder will be saved and displayed on the left side.
The relevant event queries can now be placed under the newly created folder. See 'Manage an Event Query' for more details.
Editing a query folder
- To edit the name of a query folder, select it and click the button
- Edit the name as required and click the 'Save' button
Deleting a query folder
- To delete a query folder, select it and click the button
A confirmation dialog will appear.
- Click 'Yes' in the In the confirmation dialog. Please note all event queries in the folder will also be deleted.
Event queries can be created in two ways:
An event query is built with a set of filter statements that are connected by Boolean operators, 'AND', 'OR' or 'NOT'. Each filter contains the following components.
'Field Group' + 'Field' + 'Operator + 'Value'
- Field Group - The group to which the 'Field' specified as the filter parameter belongs.
- Field - The field in the event log entry by which you want to filter results. For example, if you choose 'Agent' field group, you can select 'agent_id' or 'agent_ip' as an event field. See 'Appendix 1 – Field Groups and Event Items Description' for a full list of field groups and event field.
- Operator – Controls the relationship between the field and the specified value. Examples include 'Equals to', 'Does not equal to', contains, 'does not contain' etc.
- Value – The value for the field. Values can be entered manually or fetched from a pre-defined list which is managed in the Live List Management' interface. For example, if you choose a source IP (src_ip) as the field to be searched from network events, you can manually enter the IP address of the source of the connection request or choose a Live List containing a list of specified source IP addresses.
When the query is run, events will be fetched from the database and checked against the filter statements one by one.
Examples:
i. To search for network connection events originating from an endpoint with IP address 10.100.100.100, build the filter statement as shown below:'Source' + 'src_ip' + '=' + '10.100.100.100'
ii. To search for network connection events originating from a set of endpoints whose IP addresses start with 10.100.100.xxx, build the filter statement as shown below:
'Source' + 'src_ip' + 'AB*' + '10.100.100
iii. To search for network connection events originating from a set of endpoints whose IP addresses are defined in the 'Live List type' named 'Internal' under the 'Live List' named 'IP Blacklist', build the filter statement as shown below:'Source' + 'src_ip' + '[a]' + 'IP Blacklist' + 'Internal'
You can create more complex queries by adding more filter statements and linking them using 'AND', 'OR', or 'NOT'. For example:
- To search for network connection events originating from an endpoint with the IP 10.100.100.100, and destined for an endpoint with the IP 10.100.100.120, build the filter statements with the AND operator as shown below:
'Source' + 'src_ip' + '=' + '10.100.100.100'
AND
'Destination' + 'dst_ip' + '=' + '10.100.100.120'
To add a new event query for a customer
- Select the customer from the 'Customers' drop-down at the top of the left hand side panel.
- Select the appropriate folder or create a new query folder under which you want to create an event query. Alternatively, you can also select a folder while saving a query
- Click the button
A 'New Query' tab will be displayed.
Tip: You can also create a new query by selecting a customer then clicking the 'New Query' tab. You can save the created query by selecting an appropriate folder from the left side panel. |
The next step is to add filters to the query.
- Choose the operator for the query filter statement from the drop-down in the 'Query Builder' pane. The options available are:
- AND
- OR
- NOT
- Click the button
The 'Field Groups' drop-down and 'Fields' drop-down will appear. The 'Fields' drop-down will contain options relevant to the 'Field Group' chosen from the drop-down at the left.
- Choose the field group you wish to add to the filter from the 'Field Groups' drop-down.
The next field will display the fields available for the selected field group.
Tip: Descriptions of each Field Group and the Field items under them are available in Appendix 1 - Field Groups and Event Items Description. |
The next step is to choose the relationship operator between the two fields.
- To choose an operator, click the drop-down between the two fields:
The types operators depends on the field chosen. The following table explains the various operator symbols:
Relation Operator |
Description |
Entering the value for the 'Field' |
---|---|---|
Equals to |
|
|
Does not equal to |
|
|
Greater than |
|
|
Greater than or equal to |
|
|
Less than |
|
|
Less than or equal to |
|
|
Contains |
|
|
Does not contain |
|
|
Starts with |
|
|
Ends with |
|
|
|
Is Empty |
|
Is Not Empty |
|
|
Is in List |
Configure the filter statement to fetch values for the field from a pre-defined list containing specific values for the field type. Background:
All the values contained in the list will be included as values for the Field specified in the filter statement. |
|
Not in List |
Allows you to configure the filter statement to search for the events that do not contain specific values from a pre-defined list. On selecting as the relation parameter, drop-down options will appear for the List and the List type: The first drop-down shows the Lists that contain values for the selected query field. The second drop-down shows the List Types within the selected 'List'.
The results will display all events that do not contain the values in the lists. |
If you are adding values for source parameters like source IP address, source port, source MAC etc., but wish to reverse the parameter, click the switch icon that appears to the right of the statement. The field group and the field selected will automatically switch from source to destination or vice-versa.
- For example, if you are specifying a live list containing values of source IPs for the source IP field, but want to change them to destination IPs, you can click the switch button.
- To add a sub-filter statement, click the button beside the filter and repeat the process.
- To set the relationship between each statement, use the drop-down menu.
- For example, the query below will return events whose source ends with 10.100 OR .com AND whose destination is 86.105.227.125
Tip: You can update and refine a query by adding more filters once you have seen the results. See Updating a Query for more details. |
- To add more filter statements to the query, click the button and repeat the process.
- To delete a filter, click the button beside it.
- Click the 'Save' button in the 'Query Builder' screen.
- Enter the name of the query in the 'Query Name' field and click the 'Save' button
The 'Event Query' will be saved under the selected folder and displayed.
Note: If you didn't select a folder in the first step you will be asked to do so when saving the query. |
The next step is to run the event query. Before that, however, the 'Results' table must be checked and configured so that it is relevant to the event query. See 'Configure Results Table for a Query' and 'Event Field Selection Settings' for more details. for more details.
Creating a new query using an existing query as a template
You can select a pre-defined query and modify its parameters to create a new query.
To create a query from an existing query
- Select the customer from the 'Customers' drop-down at the top of the left hand panel.
- Select the query from the list of queries in the left panel.
The query will be expanded under a new tab in the right side panel.
-
Add or remove the query filter statements and/or edit the parameters in the existing filter statements. The process is same as creating a new condition as explained above.
- Select the folder in which the new query is to be saved.
- Click 'Save as' from the 'Query Builder' pane.
The 'Query Name' dialog will appear.
- Enter a new name for the query and click 'Save'.
The 'Event Query' will be saved and displayed under the selected folder.
The next step is to run the event query but before that the 'Results' table must be checked and configured so that it is relevant to the event query. See 'Configure Results Table for a Query' for more details.
Configure Results Table for a Query
In order to display the event fields relevant to a specific query, the 'Results' table must first be configured.
- By default, SOCaaP ships with ten event field columns in the results table.
- You can add more event field columns here.
- Select an event query from the left and click the button in the 'Query Builder' pane. Note – The event field columns added to the results are valid for this search only. Go to 'Investigation' > 'Event Field Selection Settings' to configure fields that are valid for all query searches.
The 'Result Fields Selection' dialog will be displayed.
The same 'Field Groups' and 'Fields' used for in the 'Query Builder' will be available for inclusion in the results table. By default a set of 'Result Fields' relevant to the query will be displayed.
- To add new 'Result Fields', click the 'Field Groups' combo box and select the field group.
The next field will display the items available for the selected field group.
- Select the required field from the drop-down and click the button.
- Enter a name for the field, by which the field should be displayed in the 'Results' screen.
- Repeat the process to add more fields and click 'OK'
- To remove irrelevant fields, click the trash can icon beside it.
- Click the 'Ok' button
- Click the 'Cancel' button to revert the changes you made.
- Click the 'Save' button in the 'Query Builder' screen to save your changes.
Configure Duration Based Alerts
- SOCaaP dynamically monitors customer networks for events based on used-defined queries.
- You can create queries to generate alerts if the number of events exceeds or falls below a certain threshold in a certain time-period.
- Examples. You can request alerts if the number of events matching a query exceeds 1000 in 10 minutes, or if no events are detected for a query for 15 minutes.
- Alerts can be configured to send notification emails to the admin and/or generate an 'Incident'. The incident can be auto-assigned to a user. See Manage Incidents, for more details on this.
To schedule a query to generate alerts
- Select a saved event query from the left side and click the 'Schedule' button from the 'Query Builder' pane.
The 'Schedule Info' dialog will be displayed for the selected query.
- Name – Enter a name to identify the schedule
- Description – Enter a short description for the schedule
- Duration – Enter the time period specified for monitoring the number of events matching the query, in minutes.
- Severity – Select the severity level for the alert to be generated by the schedule
- Activation – Specify whether the schedule is to be activated or not from the drop-down. You can switch the activation state of a schedule at any time from the 'Schedule Info' dialog.
- Count – Set the threshold for the number of events.
- > - Will generate an alert if the number of events detected in the specified 'duration' exceeds the value in the text field.
- < - Will generate an alert if the number of events detected in the specified 'duration' is lower than the value in the text field
- To generate an alert if no events are detected within the specified duration, choose less than and enter 'zero' ('<' and '0')
- Action – Choose how SOCaaP should react if the alert's conditions are triggered.
- Send e-mail – SOCaaP sends a notification email to the administrator if the conditions are met
- Create Incident – An incident is created and assigned to a user to investigate. See Manage Incidents for more details.
- Click 'Save' in the Schedule Info dialog to save the schedule.
The event query will be added with a schedule.
-
To edit a schedule of a query or switch the schedule between 'active' and 'inactive' states, select the query from the list at the left, click the 'Schedule' button, change the values in the 'Schedule Info' dialog and click 'Save'.
Saved event queries can be run at anytime to obtain a list of matching events within a chosen period of time. The results can be viewed in two ways:
-
As a results table with columns selected as explained above.
You can view the full details of any event in its 'Details Pane' containing values for all the fields in the log entry. The details pane also lets you add values of live lists for use in new event queries and rules. You can also run look-ups of IP addresses and domains involved in the event. More explanations are available under 'View Results Table'. -
As aggregated results.
Identified events are grouped based on selected event field(s) and the resultant event groups ranked based on the aggregation function. More explanations are available under View Aggregated Results.
To run an event query
- Select an event query from the left.
- Select the period for which you want to run the query.
- View recent events - Select a period from the drop-down at the bottom right of the 'Query Builder' pane and click 'Search'. Options range from the past hour to the past 7 days.
- View events over specific dates - Click the calendar button, enter the start and end dates and click 'Search'.
The 'Results' are displayed in the lower pane.
- Select the 'Live' check box to search streaming data for the event query.
Note: The 'Live' option is not be available for time range selections with specific start and end dates. |
The lower pane has two tabs:
- Results – The 'Results' tab displays log entries that match the query with the selected event fields as column headers (explained above). Click an event to view its details. More details on the 'Results Table' are available under 'View Results Table'.
- Aggregations – The Aggregations tab allows you to group identified events and view aggregation results. More details on aggregations are available under 'View Aggregated Results'.
After running a query, the 'Results' tab is opened by default in the lower pane. You can click the 'Results' tab to view the results table if you are currently viewing the aggregation results.
The 'Results' tab contains a table of event log records that match the event query. The event fields form the table columns. Events can be created in the Query Builder pane. See 'Configure results table for a query' and 'Event Field Selection Settings' for more details.
Each page in the 'Results' table displays 20 entries. You can navigate to successive pages using the left and right arrows at the bottom-right.
- Open a specific page - Click the page number at bottom-right then enter the page number you require:
- You can view complete details of an event log entry in the results table. You can use values from the results as additional statements in your query to further refine the search.
- You can also perform IP and domain look-ups and feed these values into live lists for use in other queries and correlation rules.
- To view fields you selected earlier, click ' Selected Result Fields'
- View event details - Click the result row then the 'Details' tab.
- View second-level filtered results - Click and select the fields you want to view then click the 'Filter' tab.
- View all collected endpoint messages and activities - Click the 'Raw Log' tab.
External IP addresses and domain names are highlighted in yellow.
- Clicking on a field adds the field with its value as a filter statement to the query, enabling you to refine your search for events that contain the same value in the respective field and/or to create a new query. See explanation under Update and Refine Queries from results for more details.
- If you click the gear icon that appears when you hover your mouse over a field you will see a context sensitive menu:
From the context sensitive menu, you can:
- Perform IP lookup of external IP addresses using IPVOID
- Perform IP Address/Domain lookup using Virus Total
- Add the value to a Live List
Performing IP Lookup of External IP Addresses using IPVOID
You can view the scan report containing IP address information and IP Blacklist Report for any external IP address detected in an event. The 'Details' pane of an event query result displays the detected external IP address field in yellow, which acts as a shortcut to perform the IP Look up through IPVOID website.
To perform IP Look up of External IP address
- Click on the event involving connection to an external network or host from the results table to open its 'Details' pane.
The fields containing external IP address(es) are highlighted in yellow as shown in the example below.
- Hover the mouse cursor over the field and click on the gear icon that appears at the right.
- Click on the 'IPVoid' option.
You will be taken to the IP VOID webpage containing the scan results of the IP address.
An example is shown below.
Performing IP Address/Domain Lookup using Virus Total
You can view the IP address information/Domain information for external IP addresses/domains detected in an event from the 'Virus Total' website. The 'Details' pane of an event query result displays the fields containing external IP address/domain names field in yellow, which acts as a shortcut to perform the look up.
To perform IP/Domain Look up using Virus Total
- Click on the event involving connection to an external network or host from the results table to open its 'Details' pane.
The fields containing external IP address/Domain name are highlighted in yellow as shown in the example below.
- Hover the mouse cursor over the field and click on the gear icon that appears at the right.
- Click on the 'Virus Total' option.
You will be taken to the Virus Total web page which contains information the IP address/domain.
An example is shown below.
Adding Field values to Live Lists from Results
You can add values for certain fields detected in an event to Live Lists defined in SOCaaP, for use in other queries and correlation rules.
Background Note on Live Lists:
|
To add a field value from an event to a live list:
- Click an event on the results table to open its 'Details' pane.
- Hover the mouse cursor over the field containing the value to be added to a list and and click on the gear icon that appears at the right:
- Click on the 'Add to List' option.
The 'List Content Add' dialog will appear. The 'Value' field in the dialog is pre-populated with the chosen value.
- Select the 'Live List' and the list type to which the value is to be added, from the respective drop-downs under 'List Management'.
- Enter the date till which the value is valid in the 'Due Date' field. You can click the calendar icon at the left of the field and choose the date. On the specified date, the value will be automatically removed from the list. If you want the value to be permanently valid, select the 'Permanent' checkbox.
- Select the customer to which the value is applicable from the 'Customer' drop-down.
- Click 'Submit'.
The value will be added to the respective 'List Type' and all the queries and correlation rules in which the list is deployed, will be updated immediately.
- The 'Aggregations' tab lets you group responses from an event query.
- Events can be grouped based on values of selected fields to form event groups.
- Event groups can then be ranked based on the aggregation function selected and results can be viewed in ascending or descending order.
To view the aggregation of events
- Click the 'Aggregations' tab in the lower right pane of the 'Event Query' interface
Aggregating events involves four steps:
Step 1 – Select the event field(s) by which events should be grouped
The first step is to choose the event fields by which the events should be grouped. Event groups will be formed so that each event group will have events with same value for the selected field. If you select more than one field, the combinations of values in the selected fields will be taken into account for grouping.
To select the event field(s) for grouping
- Choose the 'Field Group' from the first drop-down under 'Event Fields'
The next drop-down will be populated with the fields belonging the chosen group
- Choose the 'Field' from the second drop-down and click the button.
- Repeat the process to add more fields for grouping.
- You can re-position fields by selecting them then clicking the up/down arrow buttons at bottom-left
Step 2 - Select the aggregation function
The event groups formed based on the fields chosen in the first step are ranked based on the function chosen from the 'Aggregation Function' drop-down. The available options are:
- Count – The event groups are ranked based on the number of events in each group.
- For example, if you choose Source IP as 'Field', then the group which contains the most events on a particular source IP will have the top rank. The group containing the least events is ranked lowest.
- You can further control how the data is displayed by modifying the 'Order By' and 'Limit' parameters.
- Sum - The event groups are ranked based on sum of values in another field that contains numerical value.
- If you choose 'Sum', you need to select another field that contains a numerical value, like 'Bytes in/out'.
- Event groups are ranked based on the sum of the values in the chosen numerical field from all the events in that group.
- For example, if we choose 'Bytes-in' as numerical value, then the system adds up the values in the 'Bytes-in' field of all events in the group and ranks the group accordingly. This will tell you which source IP has the most incoming traffic.
- The event group with the highest SUM in the 'Bytes-in' field is ranked top and vice-versa.
- Average - Similar to above.
- Event groups are ranked based on the average values of the chosen numerical field from all the events in that group.
- For example, the average of values of 'Bytes_in' field of events in the group, if we take the same example as above
- Minimum – Similar to above. The event groups are ranked based on the minimum of the values of chosen numerical field from all the events in that group.
- Maximum – Similar to above. The event groups are ranked based on the maximum of the values of chosen numerical field from all the events in that group.
To set the aggregation function
- Choose the function from the 'Aggregation Function' drop-down.
- If you choose 'Sum', 'Average', 'Maximum' or 'Minimum', then you should choose an item which is it useful to measure.
- For example, 'Bytes-in' can be measured and is suitable for the Sum, Average, Max and Min functions.
- On the other hand, there would be little value in applying these functions to destination port numbers.
Step 3 – Select the order of ranking based on how you want to see the aggregation results
You can choose how event groups should be ranked from the 'Order By' drop-down. The available options are:
- Ascending - The group with the lowest rank will be top of the list. A limit of 5 will show the 5 groups with the lowest ranks.
- Descending - The group with the highest rank will be top of the list.. A limit of 5 will show the 5 groups with the highest ranks.
Step 4 – Set the limit for number of results to be displayed
The last step is to set a limit for the number of event groups to be displayed as aggregation results in ascending or descending order as chosen in the previous step.
- To set the limit, choose/enter the number of results to be displayed, in the 'Limit' drop-down combo box.
- Click 'Submit'.
The results will be displayed in the Aggregation Results pane at the right.
To update a query
- Select the customer from the 'Customers' drop-down at the top of the left hand side panel.
- Choose the query to be updated, from the 'Queries' list at the left.
The query with its filter statements will be displayed in a new tab at the right panel
- To delete a filter, click the button beside it.
- To add a new filter, follow the process explained in Creating a New Event Query.
To refine the query by adding a new filter(s) from the results
- Run the query as explained in the section Run an Event Query
The results will be displayed in the lower pane in the 'Results' tab.
- Click on the result, to view its details, from which new filters are to be added to the query.
The 'Selected Results Fields' pane will appear for the event log entry, with values for all the fields.
- Click one or more fields from the 'Selected Results Fields' tab
These fields will be added to the 'Filter' tab.
- To save the query with the new filter, click .
- To create a new query with the existing and newly added filters, leaving the existing query unchanged, select the category folder from the left click and save the new query with a new name.
- You can select the specific time period to create a query or save to the current query.
- See 'View Results Table' for more details on 'Raw Logs'.
- You can save event queries in order to use them for other customers.
- You can export a query folder or a particular query.
- Please note - exported event queries can only be imported to their respective sections. For example, event queries exported from the reports section can only be used in the report section. Also, the values in the filter items in the exported events for tagged and list events will be set to default values.
To export a query or query folder
- Select the customer from the 'Customers' drop-down at the top of the left hand side panel.
- Choose the query or query folder to be exported, from the 'Queries' list at the left.
- Click the 'Export' button at the bottom
The saved query can be imported for use another customer account.
- You can import saved event queries to use them for other customers.
- Imported queries can be used as is or altered to suit the requirements of the customer.
- Please note - exported event queries can only be imported to their respective sections. For example, event queries exported from the reports section can only be used in the report section. Also, the values in the filter items in the exported events for tagged and list events will be set to default values.
To import a query or query folder
- Select the customer from the 'Customers' drop-down at the top of the left hand side panel for which you want to import the saved queries
- Click the 'Import' button at the bottom
-
Navigate to the location where the event query file is saved.
- Select the file and click 'Open'
The event query or event query folder will imported and will be listed under 'Imported' folder.
You can run the queries as it is or alter according to your requirement.
Exporting Query Results to a CSV file
Administrators can save the query results by exporting it as a CSV file for future use. Please note the query results available in the opened page will only be exported.
Export a query result
- Select the customer from the 'Customers' drop-down at the top of the left hand side panel.
- Choose the query whose results to be exported, from the 'Queries' list at the left.
The Query with its filter statements will be displayed in a new tab at the right panel
- Run the query as explained in the section Run an Event Query.
The Results will be displayed in the lower pane under the 'Results' tab.
- Click the 'Export' button above the results table header
The file will be downloaded to the default download folder.