Jolt changelog

Changelog History

• v0.1.1 Changes

  May 01, 2018

  String manipulation

  • leftPad and rightPad
  • trim
  • substring
  • split

  Math

  • intSubtract, doubleSubtract, and longSubtract

  Object Manipulation

  • squashNulls - works on Lists and Maps
  • recursivelySquashNulls - recursively works on Lists and Maps

  🛠 Fixes for

  • size modify function
  • JoltUtils navigate methods
  • parens in string literals

• v0.1.0 Changes

  December 09, 2016

  No other code changes.
  ⚡️ We had to move Jenkins build machines, took this opportunity to update Java, maven, dependency versions.

• v0.0.24 Changes

  October 26, 2016

  More "modify-beta" functions : "divide" and "divideAndRound".

  Turns out the "elementAt" function did not break, it was just that params in the "method signature" were flipped.

  🚀 This release is being used by Bazaarvoice in production.

• v0.0.23 Changes

  October 01, 2016

  • 🛠 Fixed bug in Jolt Sort.
  • Usability
    • Better Exceptions thrown when parsing a Json
    • Ability to specify your own ClassLoader to lookup Transform classes

  🚀 Do not use this release if you use the "modify-beta" transform.

  • More Functions for the "modify-beta" transforms, but now some are broken
    • array "avg"
    • toBoolean
    • size
    • toString()

• v0.0.22 Changes

  July 14, 2016

  Context

  Historically, Jolt has been focused on fixing / operating on the "format" of the input JSON. It did not have a good solution / Transform for modifying the actual data, namely the "right hand side" of the input.

  🔨 "modify" is the product of a bunch of refactoring of Shiftr.

  • The "left hand side" of the spec is basically Shiftr; it does a similar parallel tree walk of the input and the spec, and reuses a lot of the Shiftr wildcard logic.
  • 🚚 The "right hand side" of modify determines what will be put in the data, while the "right hand side" of shift determines where the existing data should be moved to in the new output Map.

  Note, Modify operates on the same in memory copy of the data, whereas Shift creates a new top level output Map to populate.

  Usage and Special Characters

  Modify comes in 3 "flavors" that control how it operates on the input data, both at a leaf level but also as it is walking the tree.

  • "modify-overwrite-beta" -- (always writes)
  • 0️⃣ "modify-default-beta" -- (writes when key/index is missing or the value at key/index is null)
  • "modify-define-beta" -- (writes when key/index is missing)

  The idea is to pick the base flavor you want, and then tweak modify's behavior on a case by case basis by applying node specific overrides.

  • "+key": "..." + means overwrite
  • 0️⃣ "~key": "..." ~ means default
  • "_key": "..." _ means define

  ➕ Additionally the "?" character can suppress the modify operation, to only operate if the key/index exists

  • "key?": "..." ? means only act if input contains that key/index

  Example, say we want to process a document that may or may not have an "address" subsection.

  Spec :
  {
     "address?" : { // If address exists in the input JSON, then match otherwise skip
         "~state" : "Texas" // means if "state" does not exist or is null, then make it be "Texas"
     }
  }
  

  Functions

  Everything on the "right hand side" of modify is actually a "function". In the example above the "right hand side" of "Texas" is actually the function "insert this literal value".

  Beyond that, you can invoke "named" functions by using the "=" special character.

  Example : Say the input has a list of scores, and we want the min and max values of the list.

  input :
  {
    "scores" : [4, 2, 8, 7, 5]
  }
  
  spec :
  {
        // Pull individual data out of the scores array
        "firstScore" : "=firstElement(@(1,scores))",
        "lastScore" : "=lastElement(@(1,scores))",
  
        // Assuming that the scores array is always size of 5
        "scoreAtMidPoint" : "=elementAt(@(1,scores),2)"
  }
  
  output :
  {
    "scores" : [4, 2, 8, 7, 5],
    "firstScore" : 4,
    "lastScore" : 5,
    "scoreAtMidPoint" : 8
  }
  

  Available functions:

  Existence

  • isPresent - returns if arg[0] is present
  • notNull - returns if arg[0] is not null
  • isNull - returns if arg[0] is null

  Strings

  • toUpper - returns the Uppercased version of the input
  • toLower - returns the Lowercased version of the input
  • concat - String concatenate all the supplied arguments

  Lists

  • toList - returns args as list
  • firstElement - returns first element
  • lastElement - returns last element
  • elementAt - returns element at # index

  Math

  • max(args) - returns max element from the list of args
    • supports int, long, double, and their toString() values
  • min(args) - returns min element from list of args
    • supports int, long, double, and their toString() values
  • abs - returns abs of value
    • supports int, long, double, and their toString() values
    • supports list of the same inputs, returns list

  Type Conversion

  • toInteger - returns toInteger()
  • toLong - returns toLong()
  • toDouble - returns toDouble()

  👍 Note all off the Type Conversion functions support

  • int, long, double, and their toString() values
  • list of the same inputs, returns list

  Elvis Operator

  👉 Use an Array on the "right hand side" to specify series of functions to run / try until one returns a non-Optional.absent() result.

  • "key": ["=func1", "=func2"]

  0️⃣ Purpose, allows for looking up a value, but if it is not found, applying a default.

  Example, back to the "min and maxScore" example from above,

  spec :
  {
     // if the input document did not have a "scores" entry,
     // or it was empty,
     // or it did not contain any 'numbers'  
     // then fall back to null and zero
     "maxScore" : ["=max(@(1, scores))", null],
     "minScore" : ["=min(@(1, scores)), 0]"
  }
  

  Details about the Java Implementation

  🗄 Introduced a Function interface, marked as @deprecated as a warning

  • 🚧 As it is work in progress and implementation outside Jolt is discouraged.

  🔄 Changed baseSpec#apply(...) signature to supply availability of input

  • via Optional, which is known at lower level
  • Matched signature change into Shiftr and Cardinality, now it is possible to introduce "?" into them if needed

  Beta

  🚀 The new Modify Transform in this release is sufficient for the Bazaarvoice internal project that needs it. Usecases beyond that are not fully thought out / tested.

  In general, it still feels like it needs some work / polish before we consider it done, but the ability to do Type conversions and String concatenation are compelling and often requested features.

  Plans

  Things to do to finish the "beta"

  • 👉 Make sure all the things can be "escaped".
  • 🚚 If possible, fully implement the existing behavior of the current "cardinality" and "default" transforms as functions of Modify, so they can be deprecated / removed.
    • The "default" transform is dated and clunky, as it has not been refactored and curated like Shiftr has.
  • Expand the set of built in functions.
    • min and max work on lists
    • average
    • toBoolean
    • toString
    • length of list, string
    • String join : concat is good, but it can have a fencepost problem
    • sort

  After beta

  • 👍 Allow for users to specify their own functions.
    • Tricky, as one wants to provide some guard rails, but at the same time not limit what ppl can do.
  • 👍 Allow for a "registry" of Transforms and Functions.
    • But not as a simple Static Map, as that can cause problems if to libraries internally used Jolt.
  • Maybe "backport" the functions to Shiftr to allow for some interesting use-cases.

  Even further out

  • 👍 Allow functions to be specified inline with the spec as some kind of scripting language, aka JavaScript or Groovy
    • Doubly tricky
    • Build and arrangement of Jars, so that "jolt-core" suddenly does have a Groovy dependency
    • The interface between Jolt code and the script language.

• v0.0.21 Changes

  May 10, 2016

  Primary driver is users wanting to be able to use Jolt Shiftr to process Json-LD flavored documents.

  👀 See this test for examples.
  ✅ https://github.com/bazaarvoice/jolt/blob/c8e6ce62a2ea7551333d666d16ddba9f11feb7c4/jolt-core/src/test/resources/json/shiftr/escapeAllTheThings.json

  Note that, because Java, the "backslash" as to be doubled.

• v0.0.20

  May 10, 2016

• v0.0.19 Changes

  January 04, 2016

  👀 Interesting and powerful feature; for an example see : https://raw.githubusercontent.com/bazaarvoice/jolt/f8f18e8544f7f3ffef8c5a8f9e059a844857387f/jolt-core/src/test/resources/json/shiftr/transposeNestedLookup.json

  ➕ Additionally, "fixed" an issue where Shiftr could not differentiate between something not existing or a legitimate null in the input data.s

  However this necessitated two small backward incompatibilities

  ⚡️ 1. Low level Java interfaces used by Shiftr to output data, generally got updated to have an Optional return type.

  1. If you had specs that relied upon the behavior of not passing legitimate null, then you will need to do some work.

• v0.0.18 Changes

  December 31, 2015

  • 👍 ArrayOrderObliviousDiffy works better now
  • "@" Transpose / Lookup logic now works from inside an Array reference
  • 👍 Shiftr Array logic handles negative numbers better
  • Shfrt can now pass input "null"s thru to the output

• v0.0.17 Changes

  December 28, 2015

  🚚 The "remove" transform can now fully handle arrays.

  • As top level inputs
  • Recursing thru Arrays
  • Removing specific array indices

  Shift specs can now have "." in the output path if escaped

  So that you can have output paths like "ref.local" that do not have that be nested maps.

  Example: RHS Shiftr values and how they would manifest in the ouput tree

  • "data.ref.local" --> { "data" : { "ref" : { "local" : X } } }
  • "data.ref.local" --> { "data" : { "ref.local" : X } }

  ➕ Additional Fixes

  • 🐎 Performance fix to ArrayOrderObliviousDiffy
  • ⚡️ Updated dependency versions
  • 🛠 Fixed issue with Travis CI

  Backwards incompatible Change

  The class ChainrFactory was in the wrong package; "baz a rvoice" instead of "baz aa rvoice".
  If you were referencing that class, you will need to fix you imports.