GEMFIRE ENTERPRISE DATA FABRIC™

Enterprise Data Fabric (EDF) is an in-memory data virtualization layer that sits between a clustered application and the underlying data store to provide low-latency, high scalability, high resiliency, proactive data change notification and faster access to the reference data. EDF is highly desirable for real time applications that need memory speed access to the “now” data.

In a high performance computing environment, the high data volatility and the need to take proactive business action based upon changing data makes it important for an architecture to provide features that enable applications to not only make informed decisions based upon the reference data but also provide this ability at sub-second response time.

For the last two decades or so, relational database management systems (RDBMS) have been used to manage enterprise data. However, RDBMS takes a "kitchen sink" approach to solving any problem associated with data management by bundling as part of the database engine. RDBMS is centralized and passive in nature and lacks the ability to push data to the applications. Also, databases are designed to optimize access to disk and to guarantee the transactional properties at all times. RDBMS limits the speed and scalability of a database engine in a highly distributed environment. Additionally, traditional centralized databases are simply not designed to aggregate and distribute events that are pushed at very high rates from multiple streaming data sources.

Unlike traditional data stores, EDF provides an in-memory operational data infrastructure that brings data and events into one distributed platform. Its unique ability to analyze fast moving data and route data/events to applications with guaranteed reliability, along with its inherent distributed nature, makes EDF an ideal choice for applications that demand high speed data access, reliable message delivery & routing, high availability and scalability.

Key features of EDF:

• Memory speed data access: EDF manages data in concurrent data structures in memory. This allows it to access data at memory speed without any contention issues.
• Flexibility in data storage: Data can be stored locally in process, replicated to multiple nodes, partitioned across multiple nodes, maintained both in-memory and in-disk or simply fetched lazily from a back-end data store. Where the data is located or how many copies are maintained becomes a configuration issue and is based on the requirements around performance, high availability and volume of data being managed, yet completely abstracted away from the developer. Data locality is virtualized in a data fabric.
• Simple development model: A data fabric offers a simple API to applications to fetch and store data into the fabric. This also allows storing data without worrying about headers, encoding or decoding to some intermediate format.
• Scalable: By distributing the data, the data fabric uses resources across multiple nodes. Deployments can simply add additional capacity on the fly and automatically get the data rebalanced and available for increasing load (concurrent activity or data volume).
• Transactional: The fabric can automatically participate in any ongoing transactions and ensure consistency of data across all the applications sharing the data. For instance, if two applications concurrently decide to create the same customer order and publish this to others, the conflict will be detected and handled appropriately.
• Reliable Publish-Subscribe semantics: Applications perform create, retrieve, update and delete (CRUD) operations on a local cache and the corresponding event is routed to nodes that subscribe to the data. Data objects can either be synchronously or asynchronously pushed to subscribing applications. Events are pushed to subscribers that contain the new, changed or deleted object(s). The data fabric is intelligent enough to only propagate changes to data objects or its relationships, keeping the underlying network traffic to a minimum.
• Querying: Similar to a regular database, the data in memory can be indexed and queried using SQL like syntax.
• Continuous querying: Applications register complex queries, which are queries with complex predicates, joins, etc and, unlike a regular query, are not just executed once. They become resident in the EDF and give the impression that the query is continuously running. As the data changes the continuous query engine calculates how the result set has changed, pushes the "delta" to the application and merges this with a cache result set on the application node.
• Heterogeneous language support: Application objects are automatically serialized in a neutral format within the fabric such that the same object can be de-serialized into an instance of a class in another application written using a different language (with a similar class structure).

GemFire is an EDF solution from GemStone Systems, Inc. GemFire consists of two products:

• GemFire Enterprise™ - an in-memory data caching solution that provides low-latency and near-zero downtime along with horizontal & global scalability.