visit
Today I want to share with you our experience of building an offline-first React-Native application, using Redux-like approach on our NodeJS backend server.
Indeed we just kind of leveraged so-called event-sourcing pattern in our system without any specific tools for this purpose (e.g. or ). Although you may prefer to use this tools in your project, what I’m trying to show here is that by actually understating the core principals that stand behind event-sourcing you can gain benefits of this approach even without a need to change the whole architecture of your system or incorporate some specific rocker-science technologies and tools. Let’s get started.As software engineering began to rapidly evolve, developers realized that this traditional paradigm of thinking about data is not very effective in some type of applications (say Enterprise). As a result, new architectural patterns started to emerge.
I really encourage you to get to know yourself with DDD, CQRS and Event Sourcing patterns (as they are quite related and often mentioned side by side), to get a more wide view on data processing, but in this article, I won’t dig deep into any of them. The main point of our interest here is Event sourcing and how it changes the way we think about data.
What if instead of keeping one aggregated state of the date we would record all of the actions taken on that data and post-process them later. It’s like we don’t know at the time of how exactly this action should influence the state but we totally can’t just ignore it. So we keep appending these actions to our log and use them later when we would figure out how to deal with that.
As we all know, a few years ago Facebook suggested a new approach to state managing on frontend called Flux which hardly relies on unidirectional data flow and is completely differentiates from how things previously worked (e.g. two-way data binding in Angular). And Redux enhanced these approach with some new concepts and took an honorable place in modern frontend developer’s tools set.
In order to change the state you need to dispatch an action and an action is indeed an event description, or an event itself. And as a reaction to this event we change our state accordingly. So it’s pretty much similar to event-sourcing pattern, so how Redux really relates to event-sourcing? First of all, we should mention that Redux and event-sourcing serve the same purpose — managing the transactional state. The main difference is that Redux goes a lot more further with prescriptions on how to actually do it. Redux explicitly demonstrates how reducers over actions (events) can be the single source of truth for the application state. Event sourcing on the other hand never prescribed how events should be processed, only their creation and recording.
/* We rely on thunk-middleware to deal with asynchronous actions.If you are not familiar with the concept, go check it out,it may help you to understand our action creator's structure better */export function createQuestionAnswerEvent(payload) {return (dispatch) => {const formattedEvent = {type: ActionTypes.ANSWER_QUESTION,// It's crucial to have timestamp on every eventtimestamp: new Date().toISOString(),payload};
_// Dispatch the event itself, so our regular reducers that are responsible for analytics can process it and recalculate the statistics_
dispatch(formattedEvent);
_// Dispatch the action that declares that we want to save this particular event_
dispatch({ type: ActionTypes.SAVE\_EVENT, payload: formattedEvent });
_// At some point in time we gonna send all saved events to the backend, but let's get to it later_
dispatch(postEventsIfLimitReached());
};
}
And here is the corresponding reducer, which takes every action with type SAVE_EVENT
and pushes it the branch of our state where we collect all events and which represented by a row array
export default (state = initialState, action) => {switch (action.type) {// Append event to our events logcase ActionTypes.SAVE_EVENT:return state.concat(action.payload);
_// After sending events to the backend we can clear them_
case ActionTypes.POST\_EVENTS\_SUCCESS:
return \[\];
_/\* If user wants to clear the client cache and make sure_
_that all analytics is backend based, he can press a button,_
_which will fire CLEAR\_STORAGE action \*/_
case ActionTypes.CLEAR\_STORAGE:
return \[\];
default:
return state;
}
};
Now we have saved all of the actions which user performed while he was offline. The next logical step would be to send these events to the backend, and that’s exactly what postEventsIfLimitReached
function you’ve seen before does.
_/\* If user is online perform batch events submission \*/_
if (events.length > config.EVENTS\_LIMIT && getState().connection.isConnected) {
try {
await api.events.post(events);
dispatch({ type: ActionTypes.POST\_EVENTS\_SUCCESS });
} catch (e) {
dispatch({ type: ActionTypes.POST\_EVENTS\_FAIL });
}
}
};
}
Seems nice. We got optimistic updates on our mobile app and we managed to deliver every action of a user to the backend.
export default class EventsHandler {// Initialize an intervalstart() {this.planNewRound();}
stop() {
clearTimeout(this.timeout);
}
_// Fire processing of new events every once in a while_
planNewRound() {
this.timeout = setTimeout(async () => {
await this.main();
this.planNewRound();
}, config.eventsProcessingInterval);
}
async main() {
const events = await this.fetchEvents();
await this.processEvents(events);
}
async processEvents(events) {
const metrics = await this.fetchMetricsForEvents(events);
_/\* Here we should process events somehow._
_But HOW???_
_We'll get back to it later_
_\*/_
_/\* It's critical to mark events as read after processing,_
_so we don't fetch and apply the same events every time \*/_
await Promise.all(events.map(this.markEventAsProcessed));
}
async markEventAsProcessed(event) {
event.set({ isProcessed: true });
return event.save();
}
async fetchMetricsForEvents(events) {
_/\* I removed a lot of domain-related code from this method, for the sake_
_of simplicity. What we are doing here is accumulating ids of Metrics related to every event from argument events._
_That's how we got metricsIds \*/_
return Metric.find({ \_id: { $in: metricsIds } });
}
async fetchEvents() {
return Event.find({ isProcessed: false }).limit(config.eventsFetchingLimit);
}
}
This is a very simplified version of our class but which perfectly reflects the main idea, so you just can take a grasp of it. I removed a lot of methods, checks and domain-related logic, so you can concentrate purely on events processing.
So what we are doing with this class is:
EventsHandler
class itself, but they are forbidden in reducers)Well, here is the full version of a processEvents function you’ve seen before:
await Promise.all(metrics.map(async (metric) => {
_/\* sessionMetricReducer and trainingMetricReducer are just functions that are imported from a separate repository,_
_and are reused on the client \*/_
const reducer = metric.type === 'SESSION\_METRIC' ? sessionMetricReducer : trainingMetricReducer;
try {
_/\* Beatiful, huh? :)_
_Just a line which reduces hundreds or thousands of events_
_to a one aggregate Metric (say analytical report)_
_\*/_
const newPayload = events.reduce(reducer, metric.payload);
_/\* If an event should not reflect the metric in any ways_
_we just return an initial state in our reducers, so we can_
_have this convenient comparison by link here \*/_
if (newPayload !== metric.payload) {
metric.set({ payload: newPayload });
return metric.save();
}
} catch (e) {
console.error('Error during events reducing', e);
}
return Promise.resolve();
}));
await Promise.all(events.map(this.markEventAsProcessed));
}
Here is a quick explanation of what’s going on.
sessionMetricReducer
is a function for calculation of individual user’s statistics and trainingMetricReducer
is a function for calculation of a group statistics. They both are pure as hell. We keep them in a separate repository, covered with unit tests from head to toe, and import them on a client as well. That is called code reuse :) And we’ll get back to them later.
I bet you all know how reduce
function works in JS, but here is a quick overview of whats going on here const newPayload = events.reduce(reducer, metric.payload)
.
We have an array of events
, that’s an analogue of Redux’s actions. They have a similar structure and serve the same purpose (I’ve already shown the structure of event during its creation on client in createQuestionAnswerEvent
function).
metric.payload
is our initial state, and all you have to know about it is that it’s a plain javascript object.
So the reduce
function takes our initial state and passes it with a first event to our reducer
, which is just a pure function, which calculates new state and returns it. Then reduce
takes this new state and the next event and passes them to the reducer
again. It does this till every event won’t be applied. At the end we got a completely new Metric payload, which was influenced by every single event! Excellent.
Although notation events.reduce(reducer, metric.payload)
is very concise and simple, we may have a pitfall here if one of events would be invalid. In that case we will catch an exception for the whole pack (not only for this invalid event) and won’t be able to save the result of others valid events.
for (const event of events) {try {// Pay attention to the params order.// This way we should first pass the state and after that the event itselfconst newPayload = reducer(metric.payload, event);
if (newPayload !== metric.payload) {
metric.set({ payload: newPayload });
await metric.save();
}
As you may guess, the main challenge here is to keep Metric.payload on backend and state’s branch which represents users’ statistics on client in a similar structure. That’s the only way if you want to incorporate code reuse. By the way, events are already the same, cause if you can remember we create them on a frontend and dispatch through client reducers at first and after that we send them to the server. As long as these two conditions are met we can freely reuse the reducers.
Here is a simplified version of sessionMetricReducer, so you can make sure that it is a just a plain function, nothing scary
import moment from 'moment';
/* All are pure function, that responsible for a separate partsof Metric indicators (say separate branches of state) */import { spentTimeReducer, calculateTimeToFinish } from './shared';import {questionsPassingRatesReducer,calculateAdoptionLevel,adoptionBurndownReducer,visitsReducer} from './session';
_// Pay attention here, we'll discuss this line below_
if (currentEventDate.isBefore(lastAppliedEventDate)) return state;
switch (event.type) {
case 'ANSWER\_QUESTION': {
const questionsPassingRates = questionsPassingRatesReducer(
state.questionsPassingRates,
event
);
const adoptionLevel = calculateAdoptionLevel(questionsPassingRates);
const adoptionBurndown = adoptionBurndownReducer(state.adoptionBurndown, event, adoptionLevel);
const requiredTimeToFinish = calculateTimeToFinish(state.spentTime, adoptionLevel);
return {
...state,
adoptionLevel,
adoptionBurndown,
requiredTimeToFinish,
questionsPassingRates,
timestampOfLastAppliedEvent : event.timestamp
};
}
case 'TRAINING\_HEARTBEAT': {
const spentTime = spentTimeReducer(state.spentTime, event);
return {
...state,
spentTime,
timestampOfLastAppliedEvent : event.timestamp
};
}
case 'OPEN\_TRAINING': {
const visits = visitsReducer(state.visits, event);
return {
...state,
visits,
timestampOfLastAppliedEvent : event.timestamp
};
}
default: {
return state;
}
}
}
Take a look at the check in the beginning of the function. We want to make sure that we don’t apply an event to the metric if next events have been applied already. This check helps to ignore invalid events, if one happens.
That’s a good example of principal to always keep your events immutable. If you want to store some additional information you shouldn’t add it to event, better keep it in state somewhere. That way you can rely on events as a sort of source of truth in any point of time. Also you can establish a parallel processing of events on several machines to reach a better performance.
As I mentioned before, we keep our shared reducers in a separate repository and import them to the client and to the server. So here are the real benefits that we archived by reusing the code and storing all of the events:
And this approach can really make some future issues easier to tackle. E.g. what if we want to update analytics in a web app a in real-time, all we have to do is to subscribe
to data changes on the client (the connection may be established via WebSockets for example) and whenever data changes on backend we should send a message of a specific type to notify
the client.
Issue 1. I’ve fixed a bug in calculations but the client is still using the old cached version of analytics. How to force it to use recalculated metrics from backend?
We incorporated a version
property for each metric (kind of E-TAG for those who familiar with common cache invalidation strategies). When client fetches a metric from a server we compare client’s version with server’s version and figure out which metric is more relevant. Wins a one with a higher version. So after bug fix all we have to do is to increase version
number and the client will know that his metrics are outdated.
Issue 2. I need to create entities in offline and use their ids for futher corresponding. How to deal with that?
We adopted a simple but not perfect solution of manual creation of id on the client with and make sure that we save the entity in DB with this id. But keep in mind that it’s always better to control that kind of data on backend, in case you would change the DB or say migrate from uuidV4 to uuidV5. As an option you can use temporary ids on the client and substitute them with real ids after their creation on BEIssue 3. What should I use for data persistence in RN?
We don’t use any external solutions for this purpose, because we needed to provide data security by encrypting it, and it seems like it’s far easier to implement this by yourself. But we use for asynchronous loading of initial app state. Here is how we create Redux store:
And here are main utils from offlineWorkUtils:
export function getPersistedState() {return new Promise(async (resolve) => {// A wrapping function around AsyncStorage.getItem()const encodedState = await getFromAsyncStorage(config.persistedStateKey);
if (!encodedState) resolve(undefined);
_// A wrapping function around CryptoJS.AES.decrypt()_
const decodedState = decryptAesData(encodedState);
resolve(decodedState);
});
}
if (!state || !Object.keys(state).length) return;
try {
_// A wrapping function around CryptoJS.AES.encrypt()_
const encryptedStateInfo = encryptDataWithAes(state).toString();
_// A wrapping function around AsyncStorage.setItem()_
await saveToAsyncStorage(config.persistedStateKey, encryptedStateInfo);
} catch (e) {
console.error('Error during state encryption', e);
}
}, config.statePersistingDebounceTime);
}
For large blobs (images, audio etc) we use .
Originally published at .