GDPR forget-me app (Part 1): Requirements

May 21, 2018 · 6 min read – Last updated on October 26, 2019
heroku postgresql rabbitmq real-coding spring-boot spring-integration

In preparation for the enforcement of GDPR which becomes final on the 25th of May 2018, I’m creating a simple, open source forget-me app for craftingjava.com in compliance with Art. 17 GDPR (‘right to be forgotten’).

In the first phrase I’d like to support a simple forget-me process driven by the Scatter-Gather messaging pattern using Spring Integration and RabbitMQ, although I’ve got a tons of ideas about making a generally usable, granular and flexible consent management app for fellow bloggers. Before going into coding the app instantly, in this article I just setting the initial requirements for myself and for starting a discussion if someone else is interested in having their own forget-me app.

1. Non-technical requirements

1.1. GDPR is tricky

GDPR is tricky in that sense, that you have to delete folks’ personal data from all your systems, but you still have to be able to prove that you did comply with their request. Nowadays email address is the primary handle used to identify individuals and 90% of my subscribers use Gmail, 5% use Yahoo Mail and the rest use other providers, perhaps they are self hosted. Having that said, the first requirement is to use social login backed by Google’s, Yahoo’s and Facebook’s OAuth2-secured authentication flow. The last one would probably allow those people to login in, who use a customer email address of some kind. Spring Security 5’s new OAuth2 support will be just fine for the purpose.

1.2. Handling personal data

The app cannot persist personal data, unless it’s absolutely necessary to carry the forget-me process out. Yet, it has to be able to state that the request of a certain individual has been successfully completed. By hashing (probably a SHA-2 variant will do) email addresses and creating a user profile without any personal details this requirement can be satisfied.

1.3. Life-cycle of a subscriber

Let’s think about the life-cycle of a subscriber’s subscription. Someone signs up for my newsletter, they may unsubscribe and eventually they may ask me to forget all their personal data stored. Given these three uses cases, a subscriber can be in the following states.

  • SUBSCRIBED – denotes an active subscriber, who gave their consent to receive newsletters.
  • UNSUBSCRIBED – denotes an inactive subscriber, who withdrew their consent to receive newsletters.
  • FORGET_PENDING – means that their requested their personal data to be erased with regards to a subscription
  • FORGET_COMPLETED – means that personal data of a subscription has been deleted
  • FORGET_FAILED – means that personal data couldn’t be deleted due to a technical issue

Subscriber life-cycle state changes

  1. When someone signs up to my newsletter, the forget-me app should be notified and a subscriber and a subscription with status SUBSCRIBED needs to be created there, the same applies when for unsubscribes.
  2. Upon request all systems handling personal data needs to be asked to remove them.
  3. When someone would try to login without being having a subscription, that request can be rejected.

2. UI design

Eventually these requirements boil down to a simple UI with two pages. As a back-end guy, I just prefer to do the mock-ups quickly without any fuss on a piece of paper. Yes, I know, there are tons of wireframing tools, but nothing as fast as drawing by hand with a pencil.

Login screen

After logging in, subscribers can see their current the statuses of their subscriptions and a change log.

Status screen

Provided that the currently logged in user is in a(n) (UN)SUBSCRIBED state, the Forget Me button becomes active and after a confirmation the process can be started.

Confirmation screen

3. Technical requirements

Even before getting into the technical requirements, I’d like to make that mental note to myself, that I’m not allowed to spend more that 8 hours to complete a deployable prototype based on Spring Boot 2 to Heroku.

3.1. Software stack

  • don’t expect heavy traffic, so it’s perfectly okay and it’s rather desirable that the app go into sleep after being inactive for a while. Heroku makes a Dyno sleep after 30 mins of inactivity.
  • I’ve got only a single requirement for the database: keep running costs low. I chose PostgreSQL for the prototype as Heroku provides that out of the box and other major cloud providers (eg. Amazon RDS for PostgreSQL and Google Cloud SQL for PostgreSQL) support it as well.
  • Putting a UI together is always challenging, as there are so many JS frameworks out there. To keep things simple , this is going to be a standard Spring MVC app with Thymeleaf as the template engine and VueJS to handle UI updates.
  • The app will be connected to message broker receiving subscriber update messages and emitting erasure requests. Heroku supports various RabbitMQ providers, I used CloudAMQP before, probably will stick to that.
  • Spring Integration will be used to define and execute message flows
  • In the first phrase only a single system MailerLite will be supported, but I design a plugin-able architecture to be able to add other providers easily

3.2. Message flows

There will be two distinct messages flows, one for receiving subscriber updates from other systems from through a webhook and the other one which handles deletion requests.

Inbound message flow

In the first case an external system submits a piece of JSON describing a SUBSCRIBE or an UNSUBSCRIBE event and upon receiving that, the forget-me app registers a new subscriber and a subscription specific to that data handler. In case of MailerLite, the payload looks like this. Messages transformers will be used to translate the provider specific incoming data and extract the email address and the event’s type.

Forget request-response flow

Upon requesting data erasure, the subscriber’s request will be put into a message queue and that request will be broadcast to all the registered data handlers. Thereafter they will reply back with either an ACK or a NACK and in both cases the result will be recorded.

3.3. Messages exchanged

3.3.1. Webhook

As I mentioned above, data provided through the webhook varies widely, but with provider specific message transformers they’re going to be transformed to the following generic message.

{
  "headers": {
    "event_id":"294cab6a-d915-4654-9fc9-afc6410bf62c",
    "event_time":1526892561,
    "event_type":"webhook"
  },
  "payload": {
    "data_handler_name": "mailerlite",
    "data_handler_id":"5d4ae976-e397-4e62-9458-2a0f51c8c988",
    "subscriber_email":"test@craftingjava.com",
    "subscriber_status":"SUBSCRIBED"
  }
}

3.3.2. Forget request

{
  "headers": {
    "event_id":"294cab6a-d915-4654-9fc9-afc6410bf62c",
    "event_time":1526892561,
    "event_type":"forget-request"
  },
  "payload": {
    "data_handler_name": "mailerlite",
    "subscription_id":"99d2a1a3-84d6-48fd-8ef9-e4681f743958",
    "subscriber_email":"test@craftingjava.com"
  }
}

3.3.3. Forget response

{
  "headers": {
    "event_id":"294cab6a-d915-4654-9fc9-afc6410bf62c",
    "event_time":1526892561,
    "event_type":"forget-response"
  },
  "payload": {
    "data_handler_name": "mailerlite",
    "subscription_id":"99d2a1a3-84d6-48fd-8ef9-e4681f743958",
    "acknowledged": true
  }
}

3.4. REST API

The app is going to have a REST API with which the UI will communicate and also it will be used for performing administrative tasks as well.

Method Endpoint Access level
GET /api/subscribers admin
POST /api/subscribers admin
GET /api/subscribers/{subscriber_id} admin
POST /api/subscribers/{subscriber_id}/forget admin
GET /api/subscribers/me user
POST /api/subscribers/me/forget user
GET /api/datahandlers admin
POST /api/datahandlers admin
POST /api/datahandlers/{datahandler_id}/resetkey admin
POST /webhook/{datahandler_id}/{datahandler_key} public

In the first phrase on the endpoints in italic will be implemented.

4. Project’s resources

The forget-me app has been open sourced under AGPL v3 and it’s published to GitHub. Should you encounter an issue or have an idea, submit a ticket to JIRA.

5. Next in this series

5.1. Part 2

GDPR compliant forget-me app with Spring Integration (Part 2): In and outbound messaging

This second part focuses on how to define in- and outbound messaging with Spring Integration’s AMQP support by using the new Java DSL, which is now (as of version 5) part of the core project and doesn’t have be included as a separate dependency.

5.1. Part 3

GDPR compliant forget-me app with Spring Integration (Part 3): Conditional configuration with Spring Boot 2

Altought there will be only a single adapter supporting MailerLite, the app is modular in design and adapters are activated dynamically once they’re configured.

Back to Blog