CourierClient.send() empty response with idempotencyKey when called simultaneously

Open Q&A
1

Hi, I’m using CourierClient to send a notification to a single email recipient.
My backend has 2+ nodes that process events published by a different service (Redis) in parallel (to ensure reliability) and during the event handing I want to send an email to a user. To prevent sending 2+ events I use idempotencyKey which I build in the following manner:

${timestamp.-${trigger}-${recipient}

After I call client.send() I want to query logs to ensure the email was successfully sent (check its status).
So the code looks this way:

  async notifyEmail(email: string, notification: INotification): Promise<void> {
    const sendResult = await this.client.send(
      {
        message: {
          to: {
            email,
          },
          template: notification.id,
          data: notification.notificationCustomization,
        },
      },
      {
        idempotencyKey: notification.idempotencyKey,
      },
    );

    this.logger.log(`Successfully notified ${email} email. Request: ${sendResult.requestId}`);

    const sentNotification = await this.client.getMessage(sendResult.requestId);

    if (
      !VALID_EMAIL_STATUSES.includes(sentNotification.status) ||
      sentNotification.providers.some((provider) => VALID_EMAIL_STATUSES.includes(provider.status))
    ) {
      this.logger.warn('A notification was not properly sent. Sent notification', sentNotification);
      Sentry.captureMessage('A notification was not properly sent', {
        extra: {
          email,
          notification,
          sendResult,
          sentNotification,
        },
      });
    }
  }

So the issue I found is that when client.send() is called at the same moment on 2+ backend nodes the response differs. One node gets a normal response - sendResult in my code above would receive

{ requested: "some-uuid" }

while the second node would receive an empty string: "", note it is NOT an object as in the first case.

Because of that, the code that queries notification to check its status afterward fails when I try to call client.getMessage(sendResult.requestId) because it gets undefined instead of real requestId.

One possible solution I see here is to add a random sleep timeout before triggering client.send() like this:
await sleep(randomInteger(300, 2000)); // interval of 300 ms - 2000 ms
After I added this sleep invocation the response is always in the form of { requested: "some-uuid" }

Probably this is a bug, want to hear your thoughts on this.