LiveX Framework

Objective

The objective of the LiveX framework ("lx") to make low-latency data easier to work with. It provides an integrated set of tools to preserve the velocity of data, while bridging it into high-latency uses. It also provides a way of sharing (in real-time) data between diverse language technologies (managed and native) so they can work cooperatively with one another.

LiveX creates an environment where the most appropriate language technology can be used for each component of a low-latency system. High cost tools (C, C++) can be focused on latency sensitive components, while high level tools can be used for more routine components where cost of development is of greater importance.

Speed is paramount with LiveX, and nothing is allowed to inhibit the performance of "reflex" operations. This includes raw memory access and lock-free design. LiveX focuses on performance within a single server, it does not mandate a cross-machine communication strategy.

Components

pub?w=628&h=239

Reflex

The reflex layer is an IPC mechanism, allowing data to be shared between processes in the fastest possible way. It uses both data sharing, and message passing paradigms.

A common shared memory section is created, in which several reflex agents can read/write. Reflex provides the mechanisms to manage the opposing needs of low-latency and correctness/consistency. Shared memory is used for all data.

The reflex layer is characterised by:

  • All agents are trusted, no bad guys allowed
  • No transactional consistency
  • Raw memory access allowed, without marshalling
  • High and low level language support
  • Inter-process
  • Fully concurrent (many readers, many writers)
  • ~20ns latency, very low jitter

Live

The live layer is different, and designed to perform work that can occur in human perceived time frames. The most common use is to providing easy visualisation of data for use on user-interfaces, where speed is not needed. (probably still faster than your monitor)

The live layer is characterised by:

  • Transactional consistency
  • High-level language support (eg. C#, not C)
  • SQL-like querying
  • In-process
  • No concurrency
  • ~1-10ms latency, high jitter

Entity

The entity component includes a visual designer where entities and their relationships are defined. The designer then generates code which facilitates the following functionality:

  • Reflex: entity objects (read/write), [C#, C++, C]
  • Live: entity objects (read only) [C#]
  • Database: replication of data for persistence (read at startup, write ongoing) [SQL]

Technology

The following technologies were used to build LiveX:

Inspiration roll: RX Disruptor LINQ miniLINQ

Future bindings:

  • Linux
  • Java
  • C (ASM?)
  • Python
  • Ruby