Grail service
Grail provides APIs to store and query data, such as events, logs, and metrics.
Concepts
You need to know the following concepts to work with Grail.
Grail query API
The Grail Query API is the entry point for querying data stored in Grail.
You can query data in Grail using the . To write queries, you use Dynatrace Query Language (DQL).
There are two types of operations that the Grail Query API does:
- Query execution - Lets the client start queries and fetch the results or cancel a running query.
- Language services - Support writing DQL queries and providing more syntactic information.
Query execution
Queries are executed asynchronously via REST.
The final result is guaranteed to be available for one minute after the query has finished, so a polling interval of less than one minute is strongly recommended.
Query start
The DQL query is started by sending an HTTP POST request. The query and all corresponding parameters are part of the request body in JSON. For the following requests, you have to include the request-token as query parameter to reference the started query.
Polling and long polling
You can use request-token to poll queries. The response to a polling request contains the query's status.
Long polling
Starting a query and polling offers the possibility of specifying a request-timeout. If the final result or a preview is available before the specified timeout, the API returns the response; otherwise, it returns the state of the query.
Preview
It's possible to enable previews with the initial request. The following polling responses will contain new previews if they're available. A new preview will always be whole, not as a delta to any previous preview.
Cancel
You can cancel a query using the request-token from the response to the initial request. If the query finishes, you'll get the final result as a response to the cancellation request. Otherwise, Grail will stop the execution of the query and discard the result.
Metadata
Metadata is part of the query response next to the query result. Some metadata types are always added, whereas others
must be requested explicitly by setting the enrich parameter to specify the list of the requested metadata types.
The following metadata types are provided:
| Metadata | Description |
|---|---|
grail | The Grail metadata is always added and gives meta-information about the query execution. |
metrics | Metric metadata can be requested by adding the parameter ?enrich=metric-metadata to /query:execute and /query:poll requests. Suppose the query result has metric keys (for example, dt.host.disk.free ). In that case, the response will be enriched with an extra metadata section, metrics, which has information about those metrics, such as their units and display names. |
Endpoints
These are the endpoints for starting queries and fetching the current status, including the final result and query cancellation.
| Endpoint | Description |
|---|---|
/query:execute | Starts a Grail query. |
/query:poll | Retrieves the query status and final result from Grail. |
/query:cancel | Cancels the query and returns the result if the query has finished. Otherwise, the query and its result are discarded. |
Grail language services
The language services give support when writing DQL queries.
Endpoints
| Endpoint | Description |
|---|---|
/query:verify | Verifies a DQL query without executing it. If the query has errors, the response will have corresponding notifications. |
/query:parse | Creates a tree of the canonical form of the query. Nodes contain references to the related token positions in the DQL query string. It can be used for hover effects, marking optional items, displaying canonical forms, and more. |
/query:autocomplete | Creates a list of suggestions for the query at the given cursor position. |
Filter segments
Filter segments allow you to segment data in Grail, enhancing data analytics and troubleshooting by providing the necessary focus. They use query-time filtering to limit the results of consumer queries to selected segments.
To create a filter segment, follow the steps outlined in the documentation. Once you have created a filter segment and obtained its ID, you can use it in your queries.
If the request defines multiple segments, they will be combined using the AND boolean operator, and the Grail data will be filtered accordingly. The values defined in the request will replace the variables used in the filter segments.
Permissions
You need to have the correct permissions to query data stored in Grail. All records are stored in Grail buckets, and each bucket is assigned to a table.
Bucket permissions
A typical bucket read permission looks like ALLOW storage:buckets:read;. Adding this permission grants access to all the buckets. Note that you need table permissions in addition to this to access Grail data. Without proper table permissions, it will still not be accessible.
Bucket permissions can also be conditional, depending on which buckets need to be made accessible. The following conditional permission give access to buckets with the names default_logs and default_metrics.
ALLOW storage:buckets:read WHERE storage:bucket-name IN ("default_logs", "default_metrics");
Currently, the =, IN, STARTSWITH, !=, NOT IN, NOT STARTSWITH operators are supported in conditions.
Table permissions
There are predefined tables in Grail and permissions on those tables you can define as:
ALLOW storage:metrics:read;, ALLOW storage:logs:read;, ALLOW storage:events:read;, ALLOW storage:system:read;, etc.
You can add table permissions to grant access to the table, but you need to combine them with bucket permissions to access data stored in the buckets.
A few tables have unrestricted access and don't need bucket or table permissions, such as dt.system.data_objects.
Record level permissions
You can extend the table permissions by adding conditions on some predefined fields. Adding those conditions grants access to only those records in the table that satisfy the condition.
The following conditional table permission, a.k.a. record level permission, grants access to only those records in the metrics table that have metric key dt.host.cpu.load
ALLOW storage:metrics:read WHERE storage:metric.key = "dt.host.cpu.load";
Currently, the =, IN, and STARTSWITH operators are supported in conditions.
You can join the conditions using the AND operator.
ALLOW storage:logs:read WHERE storage:log.source STARTSWITH "/var/log" AND storage:host.name IN ("host1", "host2");
If there are many statements for the same table, they're implicitly joined by OR. The following set of conditional table permissions grant access to those records in the events table that have
an event type of PROCESS_RESTART OR an event kind of DAVIS_EVENT.
ALLOW storage:events:read WHERE storage:event.type = "PROCESS_RESTART";
ALLOW storage:events:read WHERE storage:event.kind = "DAVIS_EVENT";
Unconditional table permission overrides conditional table permissions on the same table. The following set of permissions grants access to all records in the events table:
ALLOW storage:events:read WHERE storage:event.type = "PROCESS_RESTART";
ALLOW storage:events:read;
As with table permissions, you need to combine them with bucket permissions to access data stored in the buckets.
Bucket scope permissions
You can combine both bucket and record level permissions to grant access to specific records only if they are located in the designated buckets.
The following permissions will provide access to all logs in the unrestricted_logs bucket and only specific records in the restricted_logs bucket:
ALLOW storage:buckets:read;
ALLOW storage:logs:read WHERE storage:bucket-name = "unrestricted_logs";
ALLOW storage:logs:read WHERE storage:bucket-name = "restricted_logs" AND storage:dt.security_context = "TeamA";
Please note that applying a storage:bucket-name condition to table or record level permissions will function correctly only if the user has the necessary access to the specified bucket as defined by the bucket permissions.
The following permissions will not provide any access to logs because the user has no access to the unrestricted_logs or restricted_logs buckets, resulting in an empty query result:
ALLOW storage:buckets:read WHERE storage:bucket-name STARTSWITH "default"
ALLOW storage:logs:read WHERE storage:bucket-name = "unrestricted_logs";
ALLOW storage:logs:read WHERE storage:bucket-name = "restricted_logs" AND storage:dt.security_context = "TeamA";
Currently, the =, IN, STARTSWITH, !=, NOT IN, NOT STARTSWITH operators are supported in conditions on storage:bucket-name.
Fieldset
A fieldset is a collection of fields with a unique name. You define them on a bucket, table, or environment scope.
Currently, fieldsets are available only on the spans table and support two predefined fieldsets: builtin-request-attributes-spans and builtin-sensitive-spans.
While the fields in builtin-request-attributes-spans are dynamically generated through request attributes, those in builtin-sensitive-spans remain static.
These static fields are client.ip, db.connection_string, http.request.header.referer, url.full, url.query, and db.bind.parameter.
Fieldset permissions
A typical fieldset read permission looks like ALLOW storage:fieldsets:read;. Adding this permission grants access to all sensitive fields.
Fieldset permissions can also be conditional based on the buckets and tables where access to sensitive fields is required.
The following conditional permission gives access to sensitive fields on buckets with the name default_spans and sensitive_spans.
ALLOW storage:fieldsets:read WHERE storage:bucket-name IN ("default_spans", "sensitive_spans");
The following conditional permission gives access to sensitive fields on the spans table.
ALLOW storage:fieldsets:read WHERE storage:table-name = "spans";
The following conditional permission gives access to sensitive fields with the fieldset named builtin-sensitive-spans.
ALLOW storage:fieldsets:read WHERE storage:fieldset-name = "builtin-sensitive-spans";
Currently, the =, IN, STARTSWITH, !=, NOT IN, NOT STARTSWITH operators are supported in conditions.
Endpoints
| Endpoint | Permission Handling |
|---|---|
/query:execute | Requires bucket and table permissions. |
/query:poll | Permissions to execute the request apply. Only the user that started the query can poll. |
/query:cancel | Permissions to execute the request apply. Only the user that started the query can cancel. |
/query:verify | Requires table permissions. |
/query:parse | Requires table permissions. |
/query:autocomplete | Requires bucket and table permissions. |