How to avoid slowdown when using Groovy-based @Transactional components

During the investigation of a slowdown in a back-end Spring application, I found an interesting side-effect of using class-level @Transactional annotations on Groovy-based services. If you’re using bare Groovy with Spring 4.2 (or below) and concerned about your back-end’s performance, keep reading.

1. Symptoms

While stressing the back-end app with JMeter, it was obvious that it didn’t scale well with increasing number of concurrent users. Taking a bunch of thread dumps revealed that the app was DB bound, which became a bottleneck. Most of the threads were waiting for the database performing commit.

"http-nio-8080-exec-10" #89 daemon prio=5 os_prio=0 tid=0x00007f33e0009000 nid=0x3b5c runnable [0x00007f33f52b9000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
...
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2447)
...
at com.zaxxer.hikari.proxy.ConnectionProxy.commit(ConnectionProxy.java:300)
...
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:757)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:726)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:478)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:272)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy144.getMetaClass(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.nonParamCheck(PogoMetaMethodSite.java:79)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.checkCall(PogoMetaMethodSite.java:92)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:66)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:122)
at app.security.CustomAuthenticationProvider.authenticate(CustomAuthenticationProvider.groovy:39)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:156)

Had the app modified lots of data, this behavior could have been considered normal, however, most queries didn’t modify any data.

At first sight, I didn’t spot what the pattern of the issue had been. After all, TransactionInterceptor performs commit after a transactional method had successfully executed and by default Spring uses regular proxy objects to capture the end of these calls. That said, somewhere in the stack trace, there has to be a com.sun.proxy.$ProxyNN.transactionalMethod() element. In this case, method getMetaClass() behaved transactionally. If you’re familiar with Groovy, you know that this method is declared in groovy.lang.GroovyObject.

2. Analyses

Why did it trigger commits and how did it become transactional in the first place?

As the application in entirely Grooby-based, the service’s interface (UserService) inherits the methods of groovy.lang.GroovyObject and they end up in the service’s implementation (UserServiceImpl) indirectly. Although getMetaClass() is implemented by the Groovy runtime, it’s a normal public method and using a class-level @Transactional annotation on UserServiceImpl gets TransactionInterceptor applied to these built-in methods as well.

Switching debugging on reveals that.

16:54:16.171 [localhost-startStop-1] DEBUG o.s.t.a.AnnotationTransactionAttributeSource - Adding transactional method 'LanguageServiceImpl.setMetaClass' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT; ''
16:54:16.993 [localhost-startStop-1] DEBUG o.s.t.a.AnnotationTransactionAttributeSource - Adding transactional method 'UserServiceImpl.setMetaClass' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT; ''
16:54:17.118 [localhost-startStop-1] DEBUG o.s.t.a.AnnotationTransactionAttributeSource - Adding transactional method 'CompanyServiceImpl.setMetaClas' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT; ''
16:54:20.905 [localhost-startStop-1] DEBUG o.s.t.a.AnnotationTransactionAttributeSource - Adding transactional method 'CustomTermServiceImpl.setMetaClass' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT; ''
16:54:20.913 [localhost-startStop-1] DEBUG o.s.t.a.AnnotationTransactionAttributeSource - Adding transactional method 'ContentItemServiceImpl.setMetaClass' with attribute: PROPAGATION_REQUIRED,ISOLATION_DEFAULT; ''

While stressing the system with requests, it can also be seen, that upon calling these internal Groovy methods, Spring TX perform commits, which eventually degrades performance. The stack trace above was captured from the execution of the following piece of code.

@Override
Authentication authenticate(Authentication authentication) throws AuthenticationException {
  String authenticationToken = authentication.getPrincipal()
  User user = userService.findByAuthenticationToken(authenticationToken as String)
  ...
}

Before userService.findByAuthenticationToken() would have been executed, the Groovy runtime called getMetaClass() internally as part of call() in order to fetch the meta-data of the target object.

3. Workarounds

Depending on how @Transactional was applied to services there are two possible workarounds. Using method-level annotations wherever they’re needed mitigates the issue, although it’s a tedious to apply to a large number of services. The second path would be to upgrade to Spring 4.3, as it contains SPR-14095, which fixes this issue. If you cannot upgrade however, overriding AnnotationTransactionAttributeSource is a way to go.

public class GroovyAwareAnnotationTransactionAttributeSource extends AnnotationTransactionAttributeSource {

  private static final Logger log = LoggerFactory.getLogger(GroovyAwareAnnotationTransactionAttributeSource.class);

  @Override
  public TransactionAttribute getTransactionAttribute(Method method, Class<?> targetClass) {
    if (!ClassUtils.isUserLevelMethod(method)) {
      log.trace("Transaction skipped for non-userlevel method {}", method.toString());
      return null;
    }

    return super.getTransactionAttribute(method, targetClass);
  }

}

After that some manual tweaking is also necessary to replace .

<!-- Replace <tx:annotation-driven/> with manual config in order to work a Spring bug around -->
<aop:config/>

<bean id="transactionAttributeSource" class="app .core.tx.GroovyAwareAnnotationTransactionAttributeSource"/>

<bean id="transactionInterceptor" class="org.springframework.transaction.interceptor.TransactionInterceptor">
  <property name="transactionManagerBeanName" value="transactionManager"/>
  <property name="transactionAttributeSource" ref="transactionAttributeSource"/>
</bean>

<bean class="org.springframework.transaction.interceptor.BeanFactoryTransactionAttributeSourceAdvisor">
  <property name="adviceBeanName" value="transactionInterceptor"/>
  <property name="transactionAttributeSource" ref="transactionAttributeSource"/>
</bean>