How to write acceptance tests for a Facebook Messenger bot
Here at ASOS, we’ve been experimenting with using natural language understanding capabilities to build new customer experiences in voice and conversational platforms. The first product that we launched is Enki, the ASOS Fashionbot on Facebook Messenger. Check it out here!
When building a bot for Facebook Messenger (without using a third-party integration such as Dialogflow or Wit.ai) you are struck with an immediate automation testing challenge due to the constraints of their platform. When a user sends a message to your bot, Facebook will pass the message on to your webhook (an API with a POST endpoint) and expect an immediate 200 OK response. If there are any frequent delays or unsuccessful responses, Facebook will detach your webhook from your bot and treat it as ‘unreliable’.
The key issue here is that Facebook messages are required to be processed asynchronously, which means there isn’t an actual response that you want to send to your user coming back in the HTTP response. It’s simply a ‘Message Received’ acknowledgement being returned. You have to fire up a separate process to take the message and do something with it, eventually sending the response back to Facebook’s servers through their Send API.
This creates an interesting challenge: if you want to test your deployed application at the webhook level and not through Facebook’s Messenger UI (which is subject to frequent change), you can’t simply write your standard API level test where you assert against the initial API response. You have to be a bit more creative…
Fortunately, being creative is part of our ASOS DNA! Of course, there are always many solutions to this type of problem and, within my team, we had to quickly come up with one that would allow us to start writing automated service acceptance tests as soon as possible. One of our engineering principles is to ensure that our code is well-tested and dependable, following ATDD where appropriate. We have taken an iterative approach to working with this problem and now have a solution that works for us.
Here’s an overview of how our current implementation looks:
The breakdown goes like this:
- The acceptance test constructs the message in the same format that Facebook would send it and sends it to our webhook.
- The webhook receives the message from our test and fires up a new asynchronous process to handle it. At the same time, it sends a 200 OK response back to the caller to acknowledge the message is received.
- Depending on the user ID of the message ‘sender’, our dependency injection will select the correct URL and resolve our service to either send the response to Facebook or to our mocked Facebook Send API. Normally this would be a Page Scoped ID from Facebook, however we manually prefix our acceptance test users with a fixed string that we can detect when we load our dependencies.
- The response is sent to the mocked API and stored in a thread-safe dictionary for retrieval by the test agent.
- The acceptance test agent makes a GET request against our message endpoint, which internally waits for the expected number of messages to have been sent to it before returning them to the actual acceptance test.
- The test can then assert against the message/messages received to validate the behaviour or content.
In an earlier iteration, we used Azure Service Bus instead of an API and would simply write the message responses to a queue and have the test subscribe and read from that queue. However, we found it to be very unreliable in terms of latency and message ordering, with the test suite taking as long as 30 minutes to run.
What we have now, as described briefly above, so far, is much more reliable. We have built it in a way in which multiple acceptance tests can be run in parallel, by ensuring that the user ID is generated dynamically per test, prefixed with the ‘test user’ tag for easy detection. The suite takes less than two minutes to run now!
A very simplified example
To give you an idea of how this might look, I’ve put together a very over-simplified example.
Let’s start off with the acceptance test:
The basic principle demonstrated here is that the test generates a new GUID for that session and uses it, prefixed by ‘SOME_PATTERN’, to identify the request and allow retrieval of the responses from the test endpoint. The test will send the message to our deployed webhook and request the response from the test API. The main advantage of using the GUID in conjunction with the ‘SOME_PATTERN’ prefix is that each test can be run in parallel as they won’t be sharing the same user ID.
If we look now at a simplified webhook controller, you can see a way of handling the message sent from the acceptance test and forwarding the response on to our test API:
In the above, you can see how the post method of the controller hands the message over to a separate thread and returns an OK result. Within the ProcessMessage method, we would normally do something with this message to decide what to do, then build a response to send back to Facebook. In that method, you can see a very hacky/simplified way of demonstrating how you could swap out the URL of where you send the responses depending on whether the message.SenderID starts with ‘SOME_PATTERN’. In reality, we actually use a factory to build our FacebookMessageSender class and use dependency injection to build the class with the right base URL, but the underlying logic is the same.
Finally, let’s look at what our test API looks like, which replicates the Facebook Send API for posting messages to the user, but also adds the facility to retrieve these messages for the acceptance tests:
The thought behind using things like ConcurrentDictionary is to enable thread-safe access to the messages when multiple tests are running in parallel. By making this change and enabling our tests to run in parallel instead of sequentially, we managed to bring down our test suite’s execution time from around 15 minutes to less than two minutes.
Summing it all up
As mentioned at the beginning, this solution isn’t the only way of doing things, however, I feel that it is worth sharing as I couldn’t find any guidelines or other real-world examples online about writing acceptance tests against Facebook Messenger bots/applications.
Our solution has evolved from what was originally an unreliable, extremely slow test suite (taking 30 minutes to run), to something that is optimised, simple to maintain and works for us at the moment (and now takes less than two minutes!). That, of course, may change over time and we will continue to re-evaluate and improve as the need arises. Testing is of equal importance to the actual implementation and the code is treated with equal respect, scrutiny and care.
I hope this helps someone else who is in the same boat as my team and I’d love to hear your experiences working with Facebook Messenger — in particular how you test your webhooks, so leave a comment at the bottom of this post with your feedback!
If you’d like to understand more about why we’re exploring conversational and voice interfaces here at ASOS, check out
’s post here.Finally I would like to point out that we don’t collect data from our Facebook users, however we do make use of their first name and gender (if they have provided it), to help make the experience more personalised.
If you’re concerned about how ASOS handles privacy, then please check out our privacy policy and video at https://www.asos.com/privacy-policy/
Steve Withey is a Lead Software Engineer at ASOS. He is passionate about solving problems and how technology can be used to make people’s lives easier and better, always putting the customer’s needs first! As well as tech, Steve loves his family, travel, photography, food and in what little spare time there is left, a bit of casual gaming!