Diffa

Diffa detects data differences that occur as data is moved between computing systems. It is unique insofar as it operates without the need for data to be structured in identical ways or for the systems to be built on similar software architectures or technology ‘stacks’. Diffa takes a ‘semantic’, as opposed to ‘structural’ or ‘syntactical’, view of the data that it monitors to maintain synchronisation between systems.

When differences are detected and the data sources going out of sync, Diffa provides a monitoring dashboard to view issues and can trigger automatic repair and escalation procedures. Scheduled scanning of discrete systems ensures full coverage of data at rest or in motion along with many additional aspects that are unique, such as:

  • Comparison of data in motion or at rest for semantic consistency while permitting heterodox data structures and syntax in the systems being monitored/compared
  • Use of sophisticated anonymisation and streaming algorithms to enable large volumes of confidential data to be monitored while preserving confidentiality
  • Application of Merkle trees for efficient handling of very large data sets
  • Support for multiple wire and data serialisation formats.

A couple of typical use cases include

Synchronisation:

This broad use case illustrates the ability to keep, for example:

  • Two different internal systems in sync with each other
  • An internal system in sync with an external application
  • Two different external applications in sync with each other
  • Two distinct and disjunct components of the same business application in sync with each other

Consistency:

Concerns about consistency are paramount in situations like:

  • Enterprise data pipelines (e.g. Exchange fed trading, risk and clearing)
  • Regulatory checks between systems (e.g. Claims indexing versus policy repositories for underwriters)
  • Verification of correct routing of emails confirming completions of automated processes
  • Continuous integration and application development
  • Maintenance of coherent QA test environments
  • Rapid initiation of parallel run checks