1. Introduction
|
|
1.1. Events and filters
|
|
1.2. Chained computation model
|
|
1.3. Output rates and aggregation windows
|
|
1.4. Expressions
|
|
1.4.1. Type system
|
|
1.4.2. Row type metadata
|
|
1.4.3. Scalars and aggregations
|
|
1.4.4. Aggregations state representation
|
|
1.4.5. Aggregation TTL
|
|
1.4.6. Property access
|
|
1.4.7. Special expressions
|
|
1.4.8. Function call
|
|
2. Filters
|
|
2.1. *
|
Special filter that allows all records through.
|
2.2. <term>
|
Selects records where one of the current fields matches <term>.
|
2.3. <term>~<number maxEdits>
|
Selects records where one of the current fields matches <term> with at most <maxEdits> edits.
|
2.4. [<term lower> TO <term upper>]
|
Selects records where one of the current fields is between <lower> and <upper>.
|
2.5. <field>: <filter>
|
Sets the current field to <field>. Usually followed by <term> (e.g. somefield:someterm).
|
2.6. <filter> && <filter>
|
Selects the intersection of two other filters. |
2.7. <filter> || <filter>
|
Selects the union of two other filters. |
2.8. -<filter>
|
Selects the complement of another filter. |
3. Pipes
|
|
3.1. <named...> [by <named...>] [over <window>] [every <period> | at the end]
|
Transforms or aggregates records over configurable data window and output. |
3.2. <pipe> product <pipe>
|
Computes the cartesian product of the outputs from two pipes with compatible output rates. |
3.3. <pipe> union <pipe>
|
Concatenates the outputs from two pipes with compatible output rates. |
3.4. @compress <number k>, [<number k2>,] <number... y> [by <object...>]
|
Compresses the result from the previous pipe to at most k most important rows. |
3.5. @debounce <period> [by <object...>]
|
Only outputs rows that follows <period> time without any output. |
3.6. @filter <boolean condition>
|
Filters the results from previous pipe. |
3.7. @latest
|
Keeps the latest batch of input events and output it at the end. |
3.8. @sort <sortfield... expr>
|
Sorts the results from previous pipe. |
3.9. @throttle [<number k>, ] <period> [by <object...>]
|
Limits the output of the previous pipe to at most <k> rows per <period>. |
3.10. @top <number k>, <sortfield... expr> [by <object...>]
|
Sorts the results and gets the first k rows (possibly grouped) from previous pipe. |
3.11. @unsafe
|
Marks that any pipe executed after this must run in a non-distributed environment. |
3.12. @yield [<object expr>]
|
Extracts one field of the stream to be the output event. |
4. Operators
|
|
4.1. <object># → <number>
|
Coerces the expression to number. Shorthand to <object>:number() → <number> . |
4.2. <object>$ → <string>
|
Coerces the expression to string. Shorthand to <object>:string() → <string> . |
4.3. <number> + <number> → <number>
|
Adds two numbers. |
4.4. <string> + <string> → <string>
|
Concatenates two strings. |
4.5. <number> - <number> → <number>
|
Subtracts one number from another. |
4.6. <number> * <number> → <number>
|
Multiplies two numbers. |
4.7. <number> / <number> → <number>
|
Divides one number by another (float division). |
4.8. <number> // <number> → <number>
|
Divides one number by another (integer division). |
4.9. <number> ** <number> → <number>
|
Raises one number to another's power. |
4.10. <number> % <number> → <number>
|
Returns the rest of the division of one number by another. |
4.11. -<number> → <number>
|
Negates one number. |
4.12. <boolean> and <boolean> → <boolean>
|
Returns the logical AND of two booleans. |
4.13. <boolean> or <boolean> → <boolean>
|
Returns the logical OR of two booleans. |
4.14. <boolean> xor <boolean> → <boolean>
|
Returns the logical XOR of two booleans. |
4.15. not <boolean> → <boolean>
|
Returns the logical NOT of a boolean. |
4.16. <object> == <object> → <boolean>
|
Checks whether two objects are equal. |
4.17. <object> != <object> → <boolean>
|
Checks whether two objects are not equal. |
4.18. <comparable> < <comparable> → <boolean>
|
Checks whether the left operand compares lesser than the right one. |
4.19. <comparable> <= <comparable> → <boolean>
|
Checks whether the left operand compares lesser than or equal to the right one. |
4.20. <comparable> > <comparable> → <boolean>
|
Checks whether the left operand compares greater than the right one. |
4.21. <comparable> >= <comparable> → <boolean>
|
Checks whether the left operand compares greater than or equal to the right one. |
4.22. <row> -> <identifier> → <object>
|
Extracts a field information from a strongly-typed row value. |
4.23. <object> ?? <object> → <object>
|
Returns the first if it is not null; otherwise, returns the second. |
4.24. <boolean> ? <object>, <object> → <object>
|
If the condition is true, returns the first object; otherwise, returns the second. |
5. Scalar Functions
|
|
5.1. <number>:abs() → <number>
|
Calculates the absolute value of a number. |
5.2. <number>:acos() → <number>
|
Returns the arc cosine of the argument to an angle in radians. |
5.3. <number>:asin() → <number>
|
Returns the arc sine of the argument to an angle in radians. |
5.4. <number>:atan() → <number>
|
Returns the arc tangent of the argument to an angle in radians. |
5.5. <number>:bytes([<number precision>]) → <string>
|
Formats a number as the best possible byte multiple. |
5.6. <number>:ceil([<number precision>]) → <number>
|
Returns the smallest number that is greatest than or equal to the argument. |
5.7. <number>:cos() → <number>
|
Returns the cosine of an angle in radians. |
5.8. <number>:dateadd(<period>) → <number>
|
Adds <period> to timestamp argument. |
5.9. <number>:datefloor(<period>) → <number>
|
Rounds timestamp down to the nearest date that is divisible by <period>. |
5.10. <number>:dateformat([<string format>], [<string tz>]) → <string>
|
Formats timestamp using specified format |
5.11. <number>:datesub(<period>) → <number>
|
Sutracts <period> from timestamp argument. |
5.12. <number>:exp() → <number>
|
Calculates the exponential of a number. |
5.13. <number>:floor([<number precision>]) → <number>
|
Returns the largest number that is lesser than or equal to the argument. |
5.14. <number>:format([<string format>], [<string locale>]) → <string>
|
Formats a number according to format string and locale. |
5.15. <number>:log([<number base>]) → <number>
|
Calculates the logarithm of a number. |
5.16. <number>:pow(<number exp>) → <number>
|
Raises one number to another. |
5.17. <number>:round([<number precision>]) → <number>
|
Rounds a number to <precision> decimal places. |
5.18. <number>:select(<object... list>) → <object>
|
Selects the ith element from a list of arguments. Or null if it doesn't exist. |
5.19. <number>:sin() → <number>
|
Returns the sine of an angle in radians. |
5.20. <number>:spanend(<string>, [<string tz>]) → <number>
|
Calculates end timestamp of span based on target. |
5.21. <number>:spanstart(<string>, [<string tz>]) → <number>
|
Calculates start timestamp of span based on target. |
5.22. <number>:tan() → <number>
|
Returns the tangent of an angle in radians. |
5.23. <object>:boolean() → <boolean>
|
Converts object to boolean. |
5.24. <object>:decode(<object,object... pairs>) → <object>
|
Transforms the parameter using the translation rules defined in <pairs>. |
5.25. <object>:get(<object... keys>) → <object>
|
Much like property[keys]. Works for strings, containers and arrays. |
5.26. <object>:indexin(<object... list>) → <number>
|
Returns the first index of the value in <list>, or null if <list> does not contain it. |
5.27. <object>:isin(<object... list>) → <boolean>
|
Returns true if <list> contains the value, false otherwise. |
5.28. <object>:json() → <string>
|
Converts the object to its JSON string representation. |
5.29. <object>:keep([<number ttl>]) → <object>
|
When used in a default pipe, delays or disable (if ttl not supplied or < 0) inactive group removal. |
5.30. <object>:len() → <object>
|
Tries to get <target>'s size. Works for strings, containers and arrays. |
5.31. <object>:number() → <number>
|
Converts object to number. |
5.32. <object>:object() → <object>
|
Casts any object to its canonical object representation. |
5.33. <object>:string() → <string>
|
Converts object to string. |
5.34. <string>:contains(<string>) → <boolean>
|
Returns whether the target string contains the argument. |
5.35. <string>:dateparse([<string format>], [<string tz>]) → <number>
|
Parses timestamp using specified format |
5.36. <string>:endswith(<string>) → <boolean>
|
Returns whether the target string ends with the argument. |
5.37. <string>:format(<object... args>) → <string>
|
Uses the target string as format to arguments. |
5.38. <string>:hlleval() → <number>
|
Evaluates compressed base64 HyperLogLog data. |
5.39. <string>:indexof(<string s>, [<number fromIndex>]) → <boolean>
|
Returns the index of position of <s> inside the target string. Returns null otherwise. |
5.40. <string>:lower() → <string>
|
Converts string to lowercase. |
5.41. <string>:parse([<string format>], [<string locale>]) → <number>
|
Parses a number according to format string and locale. |
5.42. <string>:regex(<string regex>) → <row>
|
Returns a strongly typed row composed by all named groups in <regex>. |
5.43. <string>:regexfind(<string regex>, [<number|string group>]) → <string>
|
Returns the matched string by <regex> in target (or one specific group). |
5.44. <string>:regexmatch(<string regex>) → <boolean>
|
Returns true if the target matches <regex>. False otherwise. |
5.45. <string>:regexsub(<string regex>, <string replacement>) → <string>
|
Replaces all matches of <regex> in target by <replacement>. |
5.46. <string>:replace(<string from>, <string to>) → <string>
|
Replaces all instances of <from> with the string <to>. |
5.47. <string>:startswith(<string>) → <boolean>
|
Returns whether the target string starts with the argument. |
5.48. <string>:substring(<number from>, [<number to>]) → <string>
|
Returns the substring between the indices <from> and <to>. |
5.49. <string>:upper() → <string>
|
Converts string to uppercase. |
5.50. compare(<comparable a>, <comparable b>) → <number>
|
Returns a number < 0 if a < b, > 0 if a > b or 0 if a = b. |
5.51. hllmerge(<string... data>) → <string>
|
Merge many instances of compressed base64 HyperLogLog data. |
5.52. max(<comparable>, <comparable>, <comparable...>) → <comparable>
|
Returns the greatest value of all supplied arguments. |
5.53. min(<comparable>, <comparable>, <comparable...>) → <comparable>
|
Returns the least value of all supplied arguments. |
5.54. newlist(<object...>) → <object>
|
Creates a instance of java.util.List with the supplied objects. |
5.55. newmap(<object,object... pairs>) → <object>
|
Creates a instance of java.util.Map with the supplied keys and values. |
5.56. pi() → <number>
|
Returns the constant value of pi. |
5.57. random(<number min>, <number max>) → <number>
|
Returns a random value between <min> and <max>. |
5.58. random([<number max>]) → <number>
|
Returns a random value of at most <max> (1 if not defined). |
5.59. timestamp() → <number>
|
Returns the most appropriate timestamp, whether in scalar or aggregation contexts. |
6. Aggregation Functions
|
|
6.1. <aggregation object expr>:if(<boolean condition>) → <object>
|
Aggregates only events that evaluates true to <condition>. |
6.2. <aggregation object expr>:overall() → <object>
|
Merges all the results from the target aggregation. |
6.3. <aggregation object expr>:overlast(<number window>) → <object>
|
Merges the results of the last <window> aggregations. |
6.4. <aggregation object expr>:prev([<number prev>]) → <object>
|
Delays and returns the previous <number>th result from target aggregation. |
6.5. all(<boolean>) → <boolean>
|
Returns true if all ocurrences evaluate true. |
6.6. any(<boolean>) → <boolean>
|
Returns true if any ocurrence evaluates true. |
6.7. avg(<number>, [<number weight>]) → <number>
|
Calculates the (possibly weighted) average of some expression. |
6.8. dcount(<object>...) → <number>
|
Estimates the field's cardinality (distinct count) using HyperLogLog. |
6.9. describe(<aggregation object expr>) → <string>
|
Yields a string json explaining the target aggregation's inner state representation. |
6.10. first(<object>) → <object>
|
Yields the ocurrence with least timestamp. |
6.11. greatest(<object>, <comparable>) → <object>
|
Yields the greatest ocurrence in the window based on some comparable. |
6.12. hll(<number log2m>, <object>...) → <number>
|
Similar to dcount, but allows configuration of log2m parameter. |
6.13. hllmerge(<string>...) → <string>
|
Performs union of many HyperLogLog encoded data in a window. |
6.14. hllset(<number log2m>, <object>...) → <string>
|
Similar to hll, but it doesn't evaluate final cardinality, just return the sketch data. |
6.15. join(<string>, [<string separator>], [<string lastSeparator>]) → <string>
|
Join all the strings in a window. |
6.16. last(<object>) → <object>
|
Yields the ocurrence with greatest timestamp. |
6.17. least(<object>, <comparable>) → <object>
|
Yields the least ocurrence in the window based on some comparable. |
6.18. map(<object key>, <object value>) → <object>
|
Creates a java.util.Map from all events in a window. |
6.19. max(<comparable>) → <comparable>
|
Yields the greatest ocurrence in the window. |
6.20. median(<number>, [<number weight>]) → <number>
|
Estimates the median value of the population using Count-Min Sketch. |
6.21. min(<comparable>) → <comparable>
|
Yields the least ocurrence in the window. |
6.22. pcount(<boolean>) → <number>
|
Aggregates the proportion of events that evaluate true to expression. |
6.23. quantile(<number q>, <number>, [<number weight>]) → <number>
|
Estimates the q (0..1) quantile of the population using Count-Min Sketch. |
6.24. set(<object>) → <object>
|
Creates a java.util.Set from all events in a window. |
6.25. smooth(<aggregation number expr>, [<number alpha>], [<number beta>]) → <number>
|
Smoothes the curve of another aggregation. |
6.26. stdev(<number>, [<number weight>]) → <number>
|
Calculates the (possibly weighted) standard deviation of some expression. |
6.27. sum(<number>) → <number>
|
Sums all evaluations of some expression. |
6.28. variance(<number>, [<number weight>]) → <number>
|
Calculates the (possibly weighted) variance of some expression. |
6.29. when(<boolean expr>) → <number>
|
Yields the latest timestamp inside window when some condition was true. |
6.30. whenfirst(<boolean expr>) → <number>
|
Yields the first timestamp inside window when some condition was true. |
6.31. Window Meta-aggregations
|
|
6.31.1. WCOUNT()
|
Yields how many outputs are merged in the current window. |
6.31.2. WSTART()
|
Yields the minimum allowed timestamp or item for the current window. |
6.31.3. WEND()
|
Yields the maximum allowed timestamp or item for the current window. |
6.31.4. OSTART()
|
Yields the minimum allowed timestamp or item for the current output. |
6.31.5. OEND()
|
Yields the maximum allowed timestamp or item for the current output. |
6.31.6. OTIMESTAMP()
|
Yields the timestamp when the output was merged (useful for item batch pipes). |
7. Timespan Language
|
|
7.1. Period definitions
|
|
7.2. Span definitions
|
|
7.2.1. now|none
|
(point, relative) Returns the reference timestamp. |
7.2.2. today
|
(interval, relative) Equivalent to current day |
7.2.3. <year>-<month>[-<day> [<hour>:[<minute>:[<second>]]]]
|
(interval, fixed) Returns the interval relative to the selected date. |
7.2.4. timestamp|ts <number>
|
(point, fixed) Returns the point with the speficied timestamp. |
7.2.5. from|since <span> to|until <span>
|
(<both>, <both>) Returns a span from the beginning of the first span to the end of the second. |
7.2.6. last <period...>
|
(interval, relative) Equivalent to <period...> before now . |
7.2.7. current|this <period>
|
(interval, relative) Returns the full period enclosing the reference timestamp. |
7.2.8. previous|yester <period>
|
(interval, relative) Returns the period before the current <period> |
7.2.9. [<period...>] before <span>
|
(<both>, <both>) The full selected period ending in the beginning of the referenced span. |
7.2.10. [<period...>] after <span>
|
(<both>, <both>) The full selected period starting at the end of the referenced span. |
7.2.11. <ordinal> <period> [of] <span>
|
(interval, <both>) Selects the nth period inside some span. |
7.2.12. <period...> ago
|
(point, relative) Select the exact timestamp of the defined period in the past. |
7.2.13. <period> of <span>
|
(interval, <both>) Selects the full period enclosing the referenced span |
7.2.14. <span> shifted by <period...>
|
(<both>, <both>) Shifts the selected span by <period> in the past. |
7.2.15. <span> shifted to <span>
|
(<both>, <both>) Calculates the span changing the reference using another span. |
7.2.16. <span> extend [left|right] [by] <number>%
|
(<both>, <both>) Extends either the start or the end of a span by a percentual value. |
7.2.17. <span> align [left|right] to <span ref>
|
(<both>, <both>) Aligns the span's total milliseconds at the left or right side of <ref> |