Posted on
by

I gave in and changed my event forwarding method in node-red for the Elite Dangerous Journal. This file is updated on various in-game events but in a way that makes it difficult to get new events only since last update. Another problem is that it’s not really a valid JSON file because it has one JSON per line but it’s not a valid JSON array. This is why it has to be parsed line by line and mashed together by event type (name) again to get the latest data for each event type per dump. Each event has it’s own timestamp by set by the game. The latest timestamp is now saved on the special flow const so node-red keeps the value in the “global” memory of the current flow:

msg.payload.event = "Journal";

let newJournalTimestamp = flow.lastJournalTimestamp;

Object.keys(msg.payload).forEach((key) => {
  if (msg.payload[key].timestamp) {
    const keyTimestamp = new Date(msg.payload[key].timestamp).getTime();

    if (!flow.lastJournalTimestamp || flow.lastJournalTimestamp < keyTimestamp) {
      // this entry is new - keep it. MULTIPLE events may have the
      //  same timestamp so wait with reassigning so we don't skip
      //  em or get the latest a 2nd time if nothing else changes.

      // update the next latest timestamp if this is newer
      if(!newJournalTimestamp || newJournalTimestamp < keyTimestamp) {
        newJournalTimestamp = keyTimestamp;
      }
    } else {
      // lastJournalTimestamp is newer, skip this
      msg.payload[key] = null;
    }
  }
});

// make sure this is a valid date for the next time
flow.lastJournalTimestamp = newJournalTimestamp || new Date().getTime();

// remove all nulled events from the payload
msg.payload = Object.fromEntries(
  Object.entries(msg.payload).filter(([_, p]) => p !== null)
);

msg.payload.timestamp = new Date(flow.lastJournalTimestamp);

return { payload: msg.payload };

So I do now keep track of the last read timestamp and reject every event that is older than the last read keeping the Journal dump smaller. This way I don’t have to try to keep track of the “latest” event to drag data from. Refuelling e.g. can happen from whopping 4 (or more) different events and it’s painful to compare all and check which one is the latest to keep track of the real current fuel levels for each tank.

Downside is I won’t get a full set of data for the current session any more if I have to reload my HUD app. This could be mitigated by using MQTT though where I could simply persist each event topic. That is already implemented and I can choose between SocketIO or MQTT in my app anyway.

4 thoughts on “Parsing the Elite Dangerous Journal”

  2. @beko that’s usually called an NDJSON file, if you didn’t know. Newline delimited json

    • @tomasekeli @beko The term I see for it is JSONL (JSON, Linebreak-delimited).

      The point is that it updates live with the game events, so it *can't* be an array of lines, as that would require replacing the closing "]" with ", { … }]" each time.

      The documentation for the Journal (and associated files) is quite clear:

      "The Player Journal is written in line-delimited JSON format (see json.org and jsonlines.org), …".

      But, yeah, after any restart of code you need to remember where you left off parsing a file, and otherwise it's best to 'tail' the file, not repeatedly open and parse the whole thing.

      • @AthanSpod @tomasekeli @beko I don't see this available for node-red. I'd love to be corrected on this. My plumbing pipeline is currently "Watch file changes /foo/bar/*" and this gets parsed as JSON line by line (so your NDJSON/JSONL).

        What I don't see is an option to "tail" such a file in node-red so I only get the latest line by default.

Reposts

Likes

Leave a Reply

To respond on your own website, enter the URL of your response which should contain a link to this post's permalink URL. Your response will then appear (possibly after moderation) on this page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Find out more about Webmentions.)