Storage service
The storage services provide APIs to store and query data, such as events, logs, and metrics.
Concepts
You need to know the following concepts to work with the storage service.
Grail query API
The Grail Query API is the entry point for querying data stored in Grail.
You can query data in Grail using the . To write queries, you use Dynatrace Query Language (DQL).
There are two types of operations that the Grail Query API does:
- Query execution - Lets the client start queries and fetch the results or cancel a running query.
- Language services - Support writing DQL queries and providing more syntactic information.
Query execution
Queries are executed asynchronously via REST.
The final result is guaranteed to be available for one minute after the query has finished, so a polling interval of less than one minute is strongly recommended.
Query start
By sending an HTTP POST request, the query is started. The query and all corresponding parameters are part of the request body in JSON. The response will contain a request-token as a query identifier for the following requests.
Polling and long polling
You can use request-token to poll queries. The response for a polling request contains the status of the query.
Long polling
Starting a query and polling offer the possibility to specify a request-timeout. If the final result or a preview is available before the specified timeout, the API returns the response; otherwise, it returns the state of the query.
Preview
It's possible to enable previews with the initial request. Following polling responses will contain new previews, if available. A new preview will always be whole, not as a delta to any previous preview.
Canceling
You can cancel a query using the request-token from the response to the initial request. If the query has finished, you'll get the final result as a response to the cancel request. Otherwise, Grail will stop the execution of the query and discard the result.
Metadata
Metadata is part of the query response next to the query result. Some metadata types are always added, whereas others
must be requested explicitly by setting the enrich parameter to specify the list of the requested metadata types.
The following metadata types are provided:
| Metadata | Description |
|---|---|
grail | The Grail metadata is always added and gives meta-information about the query execution. |
metrics | Metric metadata can be requested by adding the parameter ?enrich=metric-metadata to /query:execute and /query:poll requests. Suppose the query result contains metric keys (for example, dt.host.disk.free ). In that case, the response will be enriched with an additional metadata section, metrics, which contains information about those metrics, such as their units and display names. |
Endpoints
These are the endpoints for starting queries and fetching the current status, including the final result and query cancellation.
| Endpoint | Description |
|---|---|
/query:execute | Starts a Grail query. |
/query:poll | Retrieves the query status and final result from Grail. |
/query:cancel | Cancels the query and returns the result if the query has finished. Otherwise, the query and its result are discarded. |
Grail language services
The language services provide support when writing DQL queries.
Endpoints
| Endpoint | Description |
|---|---|
/query:verify | Verifies a DQL query without executing it. If the query contains errors, the response will have corresponding notifications. |
/query:parse | Creates a tree of the canonical form of the query. Nodes contain references to the related token positions in the DQL query string. It can be used for hover effects, marking optional items, displaying canonical forms, and more. |
/query:autocomplete | Creates a list of suggestions for the query at the given cursor position. |
Permissions
You need to have the correct permissions to query data stored in Grail. All records are stored in Grail buckets and each bucket is assigned to a table.
Bucket Permissions
A typical bucket read permission looks like ALLOW storage:buckets:read;. Adding this permission grants access to all the buckets. Note that you need table permissions in addition to this to access Grail data. Without proper table permissions, it will still not be accessible.
Bucket permissions can also be conditional depending on what buckets are needed to be made accessible. The following conditional permission provides access to buckets with the name default_logs and default_metrics.
ALLOW storage:buckets:read WHERE storage:bucket-name IN ("default_logs", "default_metrics");
Currently, =, IN, STARTSWITH operators are supported in conditions.
Table Permissions
There are predefined tables in Grail, and permissions on those tables you can define as:
ALLOW storage:metrics:read;, ALLOW storage:logs:read;, ALLOW storage:events:read;, ALLOW storage:system:read;, etc.
You can add table permissions to grant access to the table, but you need to combine them with bucket permissions to access data stored in the buckets.
A few tables have unrestricted access and don't need bucket or table permissions, such as dt.system.data_objects.
Record Level Permissions
You can extend the table permissions by adding conditions on some predefined fields. Adding those conditions grants access to only those records in the table that satisfy the condition.
The following conditional table permission, a.k.a. record level permission, grants access to only those records in the metrics table that have metric key dt.host.cpu.load
ALLOW storage:metrics:read WHERE storage:metric.key = "dt.host.cpu.load";
Currently, =, IN, and STARTSWITH operators are supported in conditions.
You can join the conditions by the AND operator.
ALLOW storage:logs:read WHERE storage:log.source STARTSWITH "/var/log" AND storage:host.name IN ("host1", "host2");
If there are multiple statements for the same table, they're implicitly joined by OR. The following set of conditional table permissions grant access to those records in the events table that have
an event type of PROCESS_RESTART OR an event kind of DAVIS_EVENT.
ALLOW storage:events:read WHERE storage:event.type = "PROCESS_RESTART";
ALLOW storage:events:read WHERE storage:event.kind = "DAVIS_EVENT";
Unconditional table permission overrides conditional table permissions on the same table. The following set of permissions grants access to all records in the events table:
ALLOW storage:events:read WHERE storage:event.type = "PROCESS_RESTART";
ALLOW storage:events:read;
As with table permissions, you need to combine them with bucket permissions to access data stored in the buckets.
Fieldset
A fieldset is a collection of fields with a unique name. You define them on a bucket, table, or environment scope.
Currently, fieldsets are available only on the spans table and support two predefined fieldsets: builtin-request-attributes-spans and builtin-sensitive-spans.
While the fields in builtin-request-attributes-spans are dynamically generated through request attributes, those in builtin-sensitive-spans remain static.
These static fields are client.ip, db.connection_string, http.request.header.referer, url.full, url.query, and db.bind.parameter. Note that spans are currently available in private preview.
Fieldset Permissions
A typical fieldset read permission looks like ALLOW storage:fieldsets:read;. Adding this permission grants access to all sensitive fields.
Fieldset permissions can also be conditional based on the buckets and tables where access to sensitive fields is required.
The following conditional permission provides access to sensitive fields on buckets with the name default_spans and sensitive_spans.
ALLOW storage:fieldsets:read WHERE storage:bucket-name IN ("default_spans", "sensitive_spans");
The following conditional permission provides access to sensitive fields on the spans table.
ALLOW storage:fieldsets:read WHERE storage:table-name = "spans";
The following conditional permission provides access to sensitive fields with the fieldset named builtin-sensitive-spans.
ALLOW storage:fieldsets:read WHERE storage:fieldset-name = "builtin-sensitive-spans";
Currently, =, IN, and STARTSWITH operators are supported in conditions.
Endpoints
| Endpoint | Permission Handling |
|---|---|
/query:execute | Requires bucket and table permissions. |
/query:poll | Permissions to execute the request apply. Only the user that started the query can poll. |
/query:cancel | Permissions to execute the request apply. Only the user that started the query can cancel. |
/query:verify | Requires table permissions. |
/query:parse | Requires table permissions. |
/query:autocomplete | Requires bucket and table permissions. |