I gave in and changed my event forwarding method in node-red for the Elite Dangerous Journal. This file is updated on various in-game events but in a way that makes it difficult to get new events only since last update. Another problem is that it’s not really a valid JSON file because it has one JSON per line but it’s not a valid JSON array. This is why it has to be parsed line by line and mashed together by event type (name) again to get the latest data for each event type per dump. Each event has it’s own timestamp by set by the game. The latest timestamp is now saved on the special flow const so node-red keeps the value in the “global” memory of the current flow:

msg.payload.event = "Journal";

let newJournalTimestamp = flow.lastJournalTimestamp;

Object.keys(msg.payload).forEach((key) => {
  if (msg.payload[key].timestamp) {
    const keyTimestamp = new Date(msg.payload[key].timestamp).getTime();

    if (!flow.lastJournalTimestamp || flow.lastJournalTimestamp < keyTimestamp) {
      // this entry is new - keep it. MULTIPLE events may have the
      //  same timestamp so wait with reassigning so we don't skip
      //  em or get the latest a 2nd time if nothing else changes.

      // update the next latest timestamp if this is newer
      if(!newJournalTimestamp || newJournalTimestamp < keyTimestamp) {
        newJournalTimestamp = keyTimestamp;
      }
    } else {
      // lastJournalTimestamp is newer, skip this
      msg.payload[key] = null;
    }
  }
});

// make sure this is a valid date for the next time
flow.lastJournalTimestamp = newJournalTimestamp || new Date().getTime();

// remove all nulled events from the payload
msg.payload = Object.fromEntries(
  Object.entries(msg.payload).filter(([_, p]) => p !== null)
);

msg.payload.timestamp = new Date(flow.lastJournalTimestamp);

return { payload: msg.payload };

So I do now keep track of the last read timestamp and reject every event that is older than the last read keeping the Journal dump smaller. This way I don’t have to try to keep track of the “latest” event to drag data from. Refuelling e.g. can happen from whopping 4 (or more) different events and it’s painful to compare all and check which one is the latest to keep track of the real current fuel levels for each tank.

Downside is I won’t get a full set of data for the current session any more if I have to reload my HUD app. This could be mitigated by using MQTT though where I could simply persist each event topic. That is already implemented and I can choose between SocketIO or MQTT in my app anyway.