Elasticsearch query

The Elasticsearch query step allows users to execute queries against the internal Elasticsearch database. As Elasticsearch is a document-based database, only a list of individual elements can be returned.

Core features

  • Full-text search, including optional fuzziness, singular/plural handling etc.
  • Keyword search, i.e. searching for exact string matches.
  • Boolean queries, which allow multiple smaller queries to be combined.
  • Range queries, used to compare numeric or date fields. This works out of the box, as Elasticsearch automatically maps well-formed numbers and dates to the appropriate field types.
  • Vector search, useful for semantic search.
    • Ember Nexus does not yet generate vector embeddings itself, i.e. the user or some bot needs to provide vectors. It is a planned feature: Issue #110.
    • Alternatively Elasticsearch also offers a semantic search feature, where Elasticsearch handles the text-to-vector conversion itself. However, this feature seems to be restricted towards enterprise-customers only.
  • Geospatial search, i.e. searching for coordinates, lines, polygons, distances to other points, inside/outside polygon checks and more.
    • As with vector search, this requires elements to be indexed with a dedicated geo_point or geo_shape field mapping, which Ember Nexus does not yet set up automatically.

Note: Elasticsearch indices are currently managed automatically, which works for most use cases. Advanced features like vector or geospatial search need manual index setup - this is possible, just not through the Ember Nexus API yet. Support for this is planned - see ticket #477.

Step parameters

In addition to query, the step’s parameters object supports the following options:

  • nodeTypes: Optional, list of node types. If provided the search is restricted to the provided node types.
  • relationTypes: Optional, list of relation types. If provided the search is restricted to the provided relation types.
  • page: Optional, int, default is 1. The page of results to return.
  • pageSize: Optional, int, default is 25. Limits the number of results per page.
  • minScore: Optional, numeric, default is null. If set to a numeric value like 1.0, removes all search results with a lower score.

Step result

  • elements: List of search element objects returned by Elasticsearch. Each object contains:
    • id: Element id (UUID).
    • type: Type of the element.
    • metadata.score: Numeric value returned by Elasticsearch. A high score implies a high fit between the query and the element.
  • totalHits: Integer, number of total elements found by the query. Can be used to calculate how many pages of results are available. Very high numbers of results might be rounded or estimates by Elasticsearch.
  • maxScore: Maximum score achieved by any element, regardless of whether it is part of the current page of results.

Parameterized Elasticsearch queries

Elasticsearch on its own does not support parameters in their Query DSL language. However, Ember Nexus uses the Expr Lang language to integrate expressions directly into the Query DSL query. These expressions need to be defined as strings surrounded by two curly brackets, e.g. {{your_expression}}, and their result will replace the expression string inside the query. The replaced value can be a string itself, but also all other JSON data types like numbers, arrays and objects are supported.
Expressions will receive the search step’s resolved parameters object as their only input - see how parameters are resolved for where those values come from.

See Expr Lang for the language’s features and a link to its playground, and parameters and expressions for a walkthrough of how parameters flow between search steps.

See the parameterized search example below.

Examples

See writing an Elasticsearch query for a walkthrough of building and running a custom query against this step, including how to use Kibana’s console and how to parameterize queries.

Unsupported Elasticsearch features

Security aspects

This search step is limited in three ways:

  • Elasticsearch search requests are readonly by default.
  • Only elements accessible by the user can be searched.
  • Only accessible element ids are returned.

While this removes support for aggregated properties and direct property lookups, this approach also disables additional dangerous approaches like the terms lookup query and joins. Note that joins are considered to be expensive, and are therefore already disabled in normal Elasticsearch deployments.

As there is no way to manipulate or exfiltrate arbitrary data, this search step is considered safe and can therefore be executed by any user.