Testing databases with Testcontainers and Liquibase

Testcontainers make it easy to test your database integration by providing docker containers for many databases, which can be started from within your unittest. This is great, but it requires a bit of work to get started since the databases are empty. Liquibase is a tool to manage database schema changes. Combining the two is an easy way to get the right schema to use in your tests. For this post we’re going to use both frameworks to setup and test a MySQL database.

Let’s start with creating a test which uses testcontainers. First we need to add some maven dependencies.

    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>testcontainers</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>mysql</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>mysql</groupId>
      <artifactId>mysql-connector-java</artifactId>
      <version>8.0.28</version>
    </dependency>
    <dependency>
      <groupId>org.testcontainers</groupId>
      <artifactId>junit-jupiter</artifactId>
      <scope>test</scope>
    </dependency>

We’re also going to add a dependencyManagement section:

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.testcontainers</groupId>
        <artifactId>testcontainers-bom</artifactId>
        <version>1.16.3</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

That should contain all the dependencies we need. Docker also needs to be installed in order for the container to start.

Now, let’s create a test that uses a MySQL container.

@Testcontainers
public class TestcontainersTest {

    @Container
    public MySQLContainer<?> mySQLContainer = new MySQLContainer("mysql:8.0.28");

    @Test
    public void testContainer(){
        mySQLContainer.start();
        try(Connection conn = mySQLContainer.createConnection("")){
            Statement statement = conn.createStatement();
            statement.execute(mySQLContainer.getTestQueryString());
            ResultSet resultSet = statement.getResultSet();
            resultSet.next();
            int result = resultSet.getInt(1);
            System.out.println(result);
        } catch (SQLException e){
            fail(e.getMessage());
        }
    }
}

The first thing to notice is the @Testcontainers annotation. This will tell the Testcontainers framework to look for containers to start. Which brings us to the next thing, the @Container annotation. This will create a container. In this case the container will be a MySQL container, specifically running version 8.0.28.

Once the container has been created, it is not started yet. In the first line of our test, we start it by calling the .start() method. After that, we request a JDBC connection to the container, and we’re set.

But we don’t have a useful database yet. That’s where Liquibase comes in. We need to add the Liquibase dependency, create the scripts that update the database, and tell Liquibase to run those scripts.

    <dependency>
      <groupId>org.liquibase</groupId>
      <artifactId>liquibase-core</artifactId>
      <version>4.7.1</version>
    </dependency>

We’ll save the liquibase scripts in /src/main/resources/database, so we can access them from the unittest.

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
    <include file="change_001.xml" relativeToChangelogFile="true" />
</databaseChangeLog>
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
    <changeSet id="001" author="ghyze.nl">
        <createTable tableName="PERSON">
            <column name="id" type="INT">
                <constraints nullable="false" primaryKey="true" primaryKeyName="pk_person" />
            </column>
            <column name = "firstname" type="varchar(255)" />
            <column name = "lastname" type="varchar(255)" />
        </createTable>
    </changeSet>
</databaseChangeLog>

And then we add the code to execute the scripts to the unittest:

    @Test
    public void testContainer(){
        mySQLContainer.start();
        try(Connection conn = mySQLContainer.createConnection("")){
            Database database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(new JdbcConnection(conn));
            Liquibase liquibase = new Liquibase("/database/changelog.xml", new ClassLoaderResourceAccessor(), database);
            liquibase.update("test");

			// some more code
        } catch (SQLException | LiquibaseException e){
            fail(e.getMessage());
        }
    }

The liquibase.update() statement needs a context. This is a way to execute or leave out some scripts. For our test this doesn’t really matter, so we just put in “test”.

And now to proof that it works, let’s insert a record in the database and read it again:

    @Test
    public void testContainer(){
        mySQLContainer.start();
        try(Connection conn = mySQLContainer.createConnection("")){
            Database database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(new JdbcConnection(conn));
            Liquibase liquibase = new Liquibase("/database/changelog.xml", new ClassLoaderResourceAccessor(), database);
            liquibase.update("test");

            PreparedStatement preparedStatement = conn.prepareStatement("insert into PERSON values(?, ?, ?)");
            preparedStatement.setInt(1, 1);
            preparedStatement.setString(2, "john");
            preparedStatement.setString(3, "doe");
            preparedStatement.execute();

            Statement statement = conn.createStatement();
            statement.execute("select * from PERSON");
            ResultSet resultSet = statement.getResultSet();
            resultSet.next();
            int id = resultSet.getInt(1);
            String firstName = resultSet.getString(2);
            String lastName = resultSet.getString(3);
            System.out.println("id: "+id+", first name:"+firstName+", last name:"+lastName);
        } catch (SQLException | LiquibaseException e){
            fail(e.getMessage());
        }
    }

And the result should look something like this:
Testcontainers test result

Conclusion

When your system uses a database, that database should be tested. There are different ways to setup and test the database, and Testcontainers is one of them. The advantage of using testcontainers is that the test is close to how the software will run on production. The disadvantage is that these tests are slow, and should be treated more as integration tests than regular unittests. In other words, you don’t want to run these tests all the time. Also, Docker needs to be installed and running. But just starting a MySQL container isn’t enough. We can’t test an empty database. Liquibase can be used to setup the database to get it in a testable state.

Verify logback log statements

The basic idea behind unittests is that they prove that the code does what it’s expected to do. That means you have to prove that the code is executed in the first place. But sometimes the only way to prove that the code is executed, is to verify that something is logged. With the recent Log4Shell exploit in mind, we’re going to use a different framework: logback.

Initial setup

Because we want to use this technique in multiple tests, we’re going to create a separate class. This class should capture all logging events for a specific class under test, and store them for later retrieval. So let’s start

public class TestUtils {

  private final Object objectUnderTest;
  
  private final List<ILoggingEvent> loggingEvents = new ArrayList<>();
  
  public TestUtils(Object objectUnderTest){
    this.objectUnderTest = objectUnderTest;
  }
}

Notice that we set this up with an actual object instead of a class. We could just use the class instead. I have chosen to include the actual object under test, so the TestUtils class can include some reflection helpers. But that is beyond the scope of this post.

Setup logging

To start capturing the logging statements, we need to create an appender. Once the appender is created, we need to start it and configure the logger to use it. The appender has one important method: append. In our case, this method should add the loggingEvent to the list of loggingEvents. The code for setting up the logging becomes as follows:

    public void setupLogging(){
        AppenderBase<ILoggingEvent> appender = new AppenderBase<ILoggingEvent>() {
            @Override
            protected void append(ILoggingEvent iLoggingEvent) {
                loggingEvents.add(iLoggingEvent);
            }
        };
        appender.start();

        ch.qos.logback.classic.Logger logger = (ch.qos.logback.classic.Logger) LoggerFactory.getLogger(objectUnderTest.getClass());
        logger.addAppender(appender);
        logger.setLevel(Level.ALL);
    }

Find logging statements

Since we’re using this for unittests, we only expect one statement with a specific message. To get the loggingEvent, if it’s there, we check all logging events:

    public Optional<ILoggingEvent> getLoggingEventWithMessage(String message){
        return loggingEvents.stream()
                .filter(iLoggingEvent -> iLoggingEvent.getMessage().equals(message))
                .findAny();
    }

How to use

Immagine you need to test the following snippet:

public class MessageLogger {
	public void logMessage(boolean bool, String message){
		if(bool){
			LOG.info("Message is {}", message);
		}
	}
}

The test we can write is the following:

public class MessageLoggerTest {
	
	@Test
	public void logMessage(){
		MessageLogger messageLogger = new MessageLogger();
		TestUtils testUtils = new TestUtils(messageLogger);
		testUtils.setupLogging();
		
		messageLogger.logMessage(true, "Hello World!");
		
		Optional<ILoggingEvent> loggingEvent = testUtils.getLoggingEventWithMessage("Message is {}");
		assertTrue(loggingEvent.isPresent());
	}
}

Note that we query for the unformatted message. The arguments can be retrieved using ILoggingEvent.getArgumentArray().

Conclusion

Sometimes we need to verify whether a logging statements has been logged in order to prove that the code under test does what it’s supposed to do. This utility class makes it easy to do so.
We’ve seen how to add a custom appender to a logback logger that stores the received loggingEvents in a list. We have created a method to query the list for an event with a specific message.
Next we have seen an example of how to use this functionality.

Send an email with Spring Boot and Thymeleaf

Using Spring Boot and Thymeleaf, it’s easy to send HTML emails to your users. This post will show the configuration and logic to send an email. We won’t go into details on how to setup the template, that’s for another time.

Project setup

Assuming you already use Spring Boot, we need to add two dependencies for this to work. The first is spring-boot-starter-mail, for sending the email. The second is spring-boot-starter-thymeleaf for handling the HTML template that we are going to use.

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-mail</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-thymeleaf</artifactId>
		</dependency>

Email configuration

We need to configure two beans. To send the email, we’re going to use Spring’s JavaMailSender. To resolve the template. we’re going to use Thymeleaf’s TemplateResolver. Since both beans are specifically configured for sending emails, we configure them in the same class.

JavaMailSender configuration

We need to create a properties file containing the mail server configuration. Let’s put it in src/main/resources/email/emailconfig.properties

email.host=mail.server.address
email.username=username
email.password=password

Next we’ll create the EmailConfiguration class.

@Configuration
@PropertySource("classpath:/email/emailconfig.properties")
public class EmailConfiguration {

    @Value("${email.host}")
    private String host;

    @Value("${email.username}")
    private String username;

    @Value("${email.password}")
    private String password;
}

Using @PropertySource we reference the properties file we created earlier. The @Value annotation is used to reference the property in the properties file.

Let’s configure the JavaMailSender bean next.

    @Bean
    public JavaMailSender getJavaMailSender() {
        JavaMailSenderImpl mailSender = new JavaMailSenderImpl();
        mailSender.setHost(host);
        mailSender.setPort(25);

        mailSender.setUsername(username);
        mailSender.setPassword(password);

        Properties props = mailSender.getJavaMailProperties();
        props.put("mail.transport.protocol", "smtp");
        props.put("mail.smtp.auth", "true");
        props.put("mail.smtp.starttls.enable", "true");
        props.put("mail.debug", "true");

        return mailSender;
    }

This sets up a JavaMailSender configured to use the smtp protocol. It’s mostly self explanatory.

TemplateEngine configuration

To resolve and process the template, we need to configure a separate TemplateEngine. This TemplateEngine will only use one TemplateResolver. You could configure more if you need to.

    @Bean
    public TemplateEngine emailTemplateEngine(){
        final SpringTemplateEngine templateEngine = new SpringTemplateEngine();
        templateEngine.addTemplateResolver(templateResolver());
        return templateEngine;
    }

    private ITemplateResolver templateResolver() {
        final ClassLoaderTemplateResolver templateResolver = new ClassLoaderTemplateResolver();
        templateResolver.setOrder(Integer.valueOf(1));
        templateResolver.setResolvablePatterns(Collections.singleton("*"));
        templateResolver.setPrefix("/email/");
        templateResolver.setSuffix(".html");
        templateResolver.setTemplateMode(TemplateMode.HTML);
        templateResolver.setCharacterEncoding("UTF-8");
        templateResolver.setCacheable(false);
        return templateResolver;
    }
}

This TemplateResolver will look for templates in src/main/resources/email, with a name ending in .html. It will handle the file as an HTML file.

Email service

Now that the beans are configured, we’re going to create a service that actually sends the email.

@Log
@Service
public class EmailService {

    public static final String SENDER_ADDRESS = "from@email.com";
    public static final String SUBJECT = "Subject of the email";
    private final TemplateEngine templateEngine;

    private final JavaMailSender emailSender;

    @Autowired
    public EmailService(JavaMailSender emailSender, TemplateEngine emailTemplateEngine){
        this.templateEngine = emailTemplateEngine;
        this.emailSender = emailSender;
    }

    public void send(Context context, String template, String address, String senderAddress, String subject){
        try {
            final MimeMessage mimeMessage = this.emailSender.createMimeMessage();
            final MimeMessageHelper messageHelper = new MimeMessageHelper(mimeMessage, "UTF-8");
            messageHelper.setFrom(senderAddress);
            messageHelper.setTo(address);
            messageHelper.setSubject(subject);

            final String htmlContent = this.templateEngine.process(template, context);
            messageHelper.setText(htmlContent, true);

            emailSender.send(mimeMessage);
        } catch (MessagingException e){
            log.warning(e.getMessage());
        }
    }
}

One thing to note here is the Context, which contains the data Thymeleaf uses to fill out the template. The ‘template’ parameter contains the filename of the template to use, e.g. ’email.html’. This file needs to be placed where the TemplateResolver will look, in src/main/resources/email.

Spring Boot – Load users from database

This is the third article in a series on authentication with Spring Boot.
In the first article we authenticated using social networks, and allowed any user to access our application.
In the second article we used inMemoryAuthentication for users that used the login form. In essence, we hardcoded our users.
This article is about adding users to a database. We are not going to allow users to sign up, we’re just going to add the users manually.

Setup Postgres

For our user entity, we want to save the following fields:

  • username
  • password (optional)
  • role
  • email
  • name

You can add the clientIds for the social networks that you allow your users to connect with for extra security. But we won’t do that here.
But we do want to be the email to be unique. Every user must have his own email propperty.

CREATE TABLE public."user"
(
    user_name text NOT NULL,
    password text,
    role text NOT NULL,
    email text NOT NULL,
    name text,
	UNIQUE(email)
)

To test this our login later, we need to add a user. The password is BCrypt encoded for “password”.

insert into "user"
(user_name, password, role, email, name)
values
('user','$2a$10$bXetyuwpEai6LomSykjZAuQ5mxU8WqhMBXGuWYnxlveCySRlGxh2i', 'USER', 'test@example.com', 'Test User')

JPA Database access

Once we have the database in place, it would be nice to actually use it in our code. To do this, we need to configure Spring to connect to our database, create a representation of the database in our code, and create a Repository that glues it together.
First, the configuration. Since we created the database schema ourselves, we don’t want Hibernate to do that. You could set spring.jpa.hibernate.ddl-auto to ‘validate’ to make sure the schema matches your model. Also, we want to show the SQL it is executing, so it’s easier to see what’s wrong. You want to turn this off when you’re done, because it generates a lot of logging.

spring:
  jpa:
    hibernate:
      ddl-auto: none
    show-sql: true
  datasource:
    url: jdbc:postgresql://localhost:5432/database
    username: databaseuser
    password: password

For the model we’re going to use Lombok, so we don’t have to deal with the boilerplate code of getters and setters. We are using the Java Persistance API to handle the database mapping.

@Getter
@Setter
@Entity
@Table(name="user", schema = "public")
public class User {

    @Id
    private String email;

    @Column(name="user_name")
    private String userName;
    private String name;
    private String password;
    private String role;
}

We use a Repository to get the User objects from the database. Spring will do a lot of magic for us, so we only need to specify the JPA query and a method in an interface to retrieve the data.

@Repository
public interface UserRepository extends CrudRepository&lt;User, String&gt; {

    @Query("SELECT u FROM User u WHERE u.userName = :username")
    User getUserByUsername(String username);

    @Query("SELECT u FROM User u WHERE u.email = :email")
    User getUserByEmail(String email);
}

UserDetails and UserDetailsService

At this point we need to have UserDetails, and a service to get them from the database. Since we’re using bot FormLogin and OAuth2, I’ve decided to implement both UserDetails and OAuth2User in the same class. This makes things easier later on.

@Repository
public class MyUserDetails implements UserDetails, OAuth2User {

    private final User user;

    public MyUserDetails(User user){
        this.user = user;
    }

    @Override
    public Map&lt;String, Object&gt; getAttributes() {
        return Collections.emptyMap();
    }

    @Override
    public Collection&lt;? extends GrantedAuthority&gt; getAuthorities() {
        SimpleGrantedAuthority authority = new SimpleGrantedAuthority(user.getRole());
        return Collections.singletonList(authority);
    }

    @Override
    public String getPassword() {
        return user.getPassword();
    }

    @Override
    public String getUsername() {
        return user.getUserName();
    }

    @Override
    public boolean isAccountNonExpired() {
        return true;
    }

    @Override
    public boolean isAccountNonLocked() {
        return true;
    }

    @Override
    public boolean isCredentialsNonExpired() {
        return true;
    }

    @Override
    public boolean isEnabled() {
        return true;
    }

    @Override
    public String getName() {
        return user.getName();
    }
}

The service tries to load a user by username, and throws an exception when no user with that username could be found.

@Component
public class MyUserDetailsService implements UserDetailsService {

    private final UserRepository userRepository;

    @Autowired
    public MyUserDetailsService(UserRepository userRepository) {
        this.userRepository = userRepository;
    }

    @Override
    public UserDetails loadUserByUsername(String username)
            throws UsernameNotFoundException {
        User user = userRepository.getUserByUsername(username);

        if (user == null) {
            throw new UsernameNotFoundException("Could not find user");
        }

        return new MyUserDetails(user);
    }
}

Update FormLogin configuration

Previously we configured in-memory authentication. Now that we have all pieces in place to retrieve our users from the database, we need to configure it.
We need to configure the UserDetailsService.

@Bean
public UserDetailsService userDetailsService(){
	return new MyUserDetailsService(userRepository);
}

Then we’ll configure a DaoAuthenticationProvider using the UserDetailService. The passwordEncoder was already configured in the last blogpost.

@Bean
public DaoAuthenticationProvider authenticationProvider() {
	DaoAuthenticationProvider authProvider = new DaoAuthenticationProvider();
	authProvider.setUserDetailsService(userDetailsService());
	authProvider.setPasswordEncoder(passwordEncoder());
	return authProvider;
}

And then to tie it toghether, we use this AuthenticationProvider as the source of the users for the LoginForm

@Override
protected void configure(AuthenticationManagerBuilder auth) throws Exception {
	auth.authenticationProvider(authenticationProvider());
}

Update OAuth configuration

Since we now use the database to store our users, we also need to update the OAuth configuration. We need to verify whether the user who’s trying to login using OAuth is actually known to us. The key information that we can use here is the email address. That’s why there is a method to get the user by email address in the repository, which we are going to use here. If there is no user with the email address that was found in the OAuth2 principle, we throw an exception. Otherwise, we return that user.

@Bean
public OAuth2UserService&lt;OAuth2UserRequest, OAuth2User&gt; oauth2UserService() {
	DefaultOAuth2UserService delegate = new DefaultOAuth2UserService();
	return request -> {
		OAuth2User auth2User = delegate.loadUser(request);
		String email = auth2User.getAttribute("email");
		User user = userRepository.getUserByEmail(email);
		if (user != null){
			return new MyUserDetails(user);
		}
		throw new InternalAuthenticationServiceException("User not registered");
	};
}

Update WebController and frontend

Now that we have all the pieces in place to use the database for user verification, we want to use this information on our site. Since there will be problems with some login attempts (maybe the user misspelled his username), we would like to be able to show an error message on the login page. So we update the /login endpoint like this:

@RequestMapping(value = "/login")
public String login(HttpServletRequest request, Model model){
	if (request.getSession().getAttribute("error.message")!= null) {
		String errorMessage = request.getSession().getAttribute("error.message").toString();
		log.info("Error message: "+errorMessage);
		model.addAttribute("errormessage", errorMessage);
	}
	return "login";
}

On the login page, we need to add the following to display this error message:

<div class="alert alert-danger" role="alert" th:if="${errormessage}">
    <span id="user" th:text="${errormessage}"></span>
</div>

In other places we would like to get the user’s name. To do this, we need to get the principal from the authentication token.

private Optional<MyUserDetails> extractMyUserDetails(Principal principal){
	if (principal instanceof UsernamePasswordAuthenticationToken) {
		return Optional.of((MyUserDetails) ((UsernamePasswordAuthenticationToken) principal).getPrincipal());
	} else if (principal instanceof OAuth2AuthenticationToken){
		return Optional.of((MyUserDetails) ((OAuth2AuthenticationToken) principal).getPrincipal());
	}
	log.severe("Unknown Authentication token type!");
	return Optional.empty();
}

And then we get the username from the MyUserDetails class

@RequestMapping(value = "/welcome")
public String welcome(Principal principal, Model model) {
	MyUserDetails userDetails = extractMyUserDetails(principal)
			.orElseThrow(IllegalStateException::new);
	model.addAttribute("name", userDetails.getName());
	return "welcome";
}

Spring Boot login with a form

Previously I wrote about securing your application with social login. But not everybody has a social account. In this article we’re going to add formlogin to the application. Formlogin simply means that your users can log in with a username and password. We’re going to keep it as simple as possible, with in-memory authentication.

Adding users

Since we’re going to use usernames and passwords to allow users to login, we need to define a password encoder. Spring will not compare the literal password that it receives, but encodes it and then compare the encoded passwords. When the passwords are stored in the database, you don’t want to see them as plain text.

@Bean
public BCryptPasswordEncoder passwordEncoder() {
	return new BCryptPasswordEncoder();
}

The next step is to add the users. We’re going to use inMemoryAuthentication with a single user, to keep it as simple as possible.

public class WebsiteApplication extends WebSecurityConfigurerAdapter {

	// some more code

	protected void configure(final AuthenticationManagerBuilder auth) throws Exception {
		auth.inMemoryAuthentication()
				.withUser("user").password(passwordEncoder().encode("password")).roles("USER");
	}
}

Note that we’re using the passwordEncoder defined in the previous step to encode the password here. Also, we need to define a role, even though we’re not using it yet.

Allow formlogin

Now we need to tell Spring Security to allow the use of our login form.

public class WebsiteApplication extends WebSecurityConfigurerAdapter {

	@Override
	protected void configure(HttpSecurity http) throws Exception {
		SimpleUrlAuthenticationFailureHandler handler = new SimpleUrlAuthenticationFailureHandler("/");
		// @formatter:off
		http
			.authorizeRequests(
					a -&gt; a
				.antMatchers("/error", "/webjars/**","/oauth/**","/login").permitAll()
				.anyRequest().authenticated()
			)
			.formLogin(f -&gt; f
					.loginPage("/login").permitAll()
			)
			// some more code
			;
		// @formatter:on
	}
	// some more code
}

Add login form

Now that we have enabled the formlogin in the backend, it’s time to add a login form in the frontend.

<form name='loginForm' th:action="@{/login}" method='POST'>
	<table>
		<tr>
			<td>User:</td>
			<td><input type='text' name='username' value=''></td>
		</tr>
		<tr>
			<td>Password:</td>
			<td><input type='password' name='password' /></td>
		</tr>
        <tr>
            <td><input name="submit" type="submit" value="submit" /></td>
        </tr>
    </table>
</form>

Webcontroller changes

There were two methods that use the authentication information. The first, index(), doesn’t need to change at all. Sure, we get a different Authentication implementation, but it still works unaltered.

@RequestMapping(value = {"/","/index"})
public String index(@CurrentSecurityContext(expression = "authentication") Authentication authentication) {
	if (authentication.isAuthenticated()) {
		return "redirect:/welcome";
	}
	return "redirect:/login";
}

The second method, where we try to find the username of the authenticated user does change. Here we drop the @CurrentSecurityContext annotation, and request the Principal directly. However, based on the method of authentication, we need to do something else to get the username. If the user used the login form, we can get the username directly from the principal. When the user authenticated using OAuth2, we need to dig a little deeper.

@RequestMapping(value = "/welcome")
public String welcome(Principal principal, Model model) {
	if (principal instanceof UsernamePasswordAuthenticationToken) {
		model.addAttribute("name", principal.getName());
	} else if (principal instanceof OAuth2AuthenticationToken){
		model.addAttribute("name", ((OAuth2AuthenticationToken) principal).getPrincipal().getAttribute("name"));
	}
	return "welcome";
}

Conclusion

Adding a login form to a Spring Boot application is easy. We’ve hardcoded a user with in memory authentication. Since we’re going to use a password, we need to have a password encoder. We’ve configured Spring Security to allow the user to login using a form. Then we’ve actually added a basic HTML login form, and as a last step we’ve modified the backend to work with a different type of principal.

Spring Boot and Oauth2 with Thymeleaf

Spring has a good tutorial explaining how to authenticate with your application using one or more external authentication providers, like GitHub or Google. This tutorial uses a single page application with a Rest endpoint. For a personal project I didn’t want a single page application, I wanted to use Thymeleaf. During implementation I discovered a few things that I’d like to share. This post continues where the tutorial stopped, so you might want to read the tutorial first.

Configure custom OAuth2UserService

When the user has authenticated using an external service, you probably want to do something with that information. Most commonly you’d want to find the user in your own database. You need to have a hook where you can get access to the authenticated user details. To do this, you can create your own OAuth2UserService bean which will be executed when the user has been authenticated. The bean itself can be quite basic, the following example only returns the authenticated user:

 	@Bean
	public OAuth2UserService<OAuth2UserRequest, OAuth2User> oauth2UserService() {
		DefaultOAuth2UserService delegate = new DefaultOAuth2UserService();
		return request -> {
			OAuth2User user = delegate.loadUser(request);

            // custom code

			return user;
		};
	}

Now we need to tell our application when to call this bean. In the configure(HttpSecurity http) method, we’re going to add the following fragment:

			.oauth2Login(o -> o.failureHandler((request, response, exception) -> {
						request.getSession().setAttribute("error.message", exception.getMessage());
						handler.onAuthenticationFailure(request, response, exception);
					})
				.userInfoEndpoint()
				.userService(oauth2UserService())
			);

Now we’ve specifically told Spring Security to use our own OAuth2UserService.

Authenticating with Google

Using our own custom OAuth2UserService, I discovered that authenticating with Google didn’t work. Or rather, the custom OAuth2UserService wasn’t executed while it was executed when using GitHub or Facebook. It turned out that when you don’t specify which scope you’re interested in, Google returned all scopes. Included in the list was “openid”, which is specifically filtered out by Spring Security. So, if you want to use your own OAuth2UserService with Google, you need to configure it with the scopes you need. Like this:

spring:
  security:
    oauth2:
      client:
        registration:
          google:
            clientId: google-client-ID
            clientSecret: google-client-secret
            scope:
              - email
              - profile

@CurrentSecurityContext

To obtain the authenticated user principal, the tutorial uses the annotation @AuthenticationPrincipal, like this:

    @GetMapping("/user")
    public Map<String, Object> user(@AuthenticationPrincipal OAuth2User principal) {
        return Collections.singletonMap("name", principal.getAttribute("name"));
    }

Usually you’re only interested in the user. However, if you need more information, you can use @SecurityContext. This article provides more information.
Here are two examples of how to use @CurrentSecurityContext with Thymeleaf:

    @RequestMapping(value = {"/","/index"})
    public String index(@CurrentSecurityContext(expression = "authentication") Authentication authentication) {
        if (authentication.isAuthenticated()) {
            return "redirect:/welcome";
        }
        return "redirect:/login";
    }
    @RequestMapping(value = "/welcome")
    public String welcome(@CurrentSecurityContext(expression = "authentication.principal") OAuth2User user, Model model) {
            model.addAttribute("name", user.getAttribute("name"));
            return "welcome";
    }

Custom Access Denied Page

The Spring Boot tutorial throws an unauthorized exception when the user tries to access a resource that he’s not allowed to. I wanted the website to redirect to the login page, assuming the user wasn’t authenticated. Or, if he was, then he should be redirected to the welcome page. We can configure this in the configure(HttpSecurity http) method:

			.exceptionHandling(e -> e
					.accessDeniedPage("/")
			)

This redirection ends up in the index() method of the previous section. Let’s look at that method again:

    @RequestMapping(value = {"/","/index"})
    public String index(@CurrentSecurityContext(expression = "authentication") Authentication authentication) {
        if (authentication.isAuthenticated()) {
            return "redirect:/welcome";
        }
        return "redirect:/login";
    }

When the user is authenticated, the user is redirected to the welcome page. Otherwise, the user is redirected to the login page.

Conclusion

While the Spring tutorial is quite good, there’s a lot more to OAuth2 authentication than it covers. This article covers some subjects that were beyond the scope of Spring’s tutorial. We’ve seen how to add your own processing of the authenticated user. Next we discussed some quirks when authenticating with Google. Then we’ve seen an alternative and more flexible way to get access to the user details. And last we redirected the access denied page to either the login page or the welcome page, using Thymeleaf.

Apache HttpClient

For a hackaton I wanted to read some files from our BitBucket server. I knew the URLs of the files, but there were some complications. First, you need to be authenticated. According to the documentation, the preferred way of authentication is HTTP Basic Authentication when authenticating over SSL. We are using an SSL connection, but with self-signed certificates.

When working with SSL, Java uses keystores and truststores. The difference between the two is that keystores store private keys, which are used by the server side, and truststores contain public keys, which are used by the client side. We have our own custom truststore, and we can tell the JVM to use that one by passing the following parameters:

-Djavax.net.ssl.trustStore=/truststore/location -Djavax.net.ssl.trustStorePassword=password

This works when you only want to access sites using your custom truststore. As soon as you want to make a connection to public sites, this fails. By default you can only use one truststore at a time. If you want more, you have create some custom code.

Or you can use the Apache HttpClient.

HTTP Basic Authentication

There are a couple of ways to use Basic Authentication. For BitBucket we need to use Preemptive Basic Authentication, which means we need to configure a HttpClientContext.
The first thing we need to do is to setup the CredentialsProvider. This doesn’t need much explanation.

        // Configure CredentialsProvider
        final CredentialsProvider provider = new BasicCredentialsProvider();
        final UsernamePasswordCredentials credentials
                = new UsernamePasswordCredentials("username", "password");
        provider.setCredentials(AuthScope.ANY, credentials);

Next we need to configure an AuthCache. We’re going to cache the authentication for a specific host.

        // configure AuthCache
        final HttpHost targetHost = new HttpHost("host", PORT, "https");
        final AuthCache authCache = new BasicAuthCache();
        authCache.put(targetHost, new BasicScheme());

Last, we use the previous steps to configure our HttpClientContext

        // configure HttpClientContext
        final HttpClientContext context = HttpClientContext.create();
        context.setCredentialsProvider(provider);
        context.setAuthCache(authCache);

HttpClient with custom SSL context

Now it’s time to configure our HttpClient. We’re going to load our truststore specifically for this client. This means that other clients and connections will still use the default Java truststore.

        SSLContext sslContext = new SSLContextBuilder()
                .loadTrustMaterial(
                        new File(configuration.getTruststoreLocation()),
                        configuration.getTruststorePassword().toCharArray()
                ).build();

        SSLConnectionSocketFactory sslSocketFactory = 
                new SSLConnectionSocketFactory(sslContext);
        return HttpClientBuilder.create()
                .setSSLSocketFactory(sslSocketFactory)
                .build();

Using the HttpClient to execute the request

Now that we have configured both the HttpClient and its context, executing a request also becomes easy. Note that we need to pass the context to the execute method.

        final HttpGet request = new HttpGet(link);
        HttpResponse response = httpClient.execute(request, context);
        InputStream connectionDataStream = response.getEntity().getContent();

Linux: Spring Boot as a service on port 80

Well, that’s a mouth full. This blog shows how to run your Spring Boot application on port 80 when your linux server starts. We are going to use Ubuntu or Linux Mint for this, and we’re going to assume that Java is installed.

Setup Spring Boot

The first thing we need to do is tell Maven to make our Spring Boot application runnable. We’ll configure the spring-boot-maven-plugin to make the jar that’s going to be built executable.

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
				<configuration>
					<executable>
						true
					</executable>
				</configuration>
			</plugin>
		</plugins>
	</build>

By default Spring Boot runs on port 8080. There’s no need to change that here, we’ll deal with that later. But if you want, you could start the application on a different port by configuring it in application.properties

server.port=8079

Run Application as a Service

Set permissions

The first thing we need to do is to copy the application to its final destination, and create a user to run the app. By creating a separate user for this application, we limit the damage that can be done in case there is a security breach. I’m going to call the application “website” and I’m going to place it under /var/website, to make things easy. Once the application is in its place, change directory to /var/website, create a new user and transfer ownership to the new user. We’re going to allow the newly created user to read and execute the file, but not to write to the file. This, again, limits the damage that can be done when there’s a security breach.

sudo useradd website
sudo passwd website
sudo chown website:website website.jar
sudo chmod 500 website.jar

Setup systemd service

To manage starting and stopping our application, we’re going to turn it into a service maintained by systemd. We need to tell systemd how to run our service by creating a configuration file.

create /etc/systemd/system/website.service

[Unit]
Description=My spring-boot application

Wants=network.target
After=syslog.target

[Service]
User=website
Type=simple
ExecStart=/var/website/website.jar
Restart=on-failure
RestartSec=60
KillMode=mixed

[Install]
WantedBy=multi-user.target

This is what the options mean:
Description: A text description of what the service is for.
Wants: The services that should, but are not required to, be started before this service.
After: The services that should be started (if they are not already running) after this service has been started.
User: The user that should be used to start this service.
Type: This indicates the way the service is started. We’ll set it to simple.
ExecStart: The command with parameters that needs to be executed to start this service.
Restart: When the service should be restarted.
RestartSec: If the service needs to be restarted, how many seconds systemd should wait.
KillMode: How the process will be stopped by systemd. We’ll set it to mixed, which sends a SIGTERM signal to our service which allows the service to shut down itself.
WantedBy: We want our service to be started when the system is up and running, and accepting connections from users.

For a list of options, look here.
KillMode is described here.
For more info on WantedBy, see this.

For the security reasons mentioned above, we’ll change the permissions of this file:

sudo chmod 640 /etc/systemd/system/website.service

Enable service to start on boot

The next thing we want to do is to indicate that this service should be started when the server is starting.

sudo systemctl enable website

Setup nginx

Now we have our application installed as a service, and it’s started when the server starts up. But the application is running on port 8079, which is not what we want. Linux only allows the root user to open TCP ports below 1000. To fix this, we’re going to use nginx to forward port 80 to our application.

create /etc/nginx/sites-enabled/website.conf with the following content:

server {
        listen  80;
        server_name     _;
        location /  {
            proxy_pass          http://localhost:8079/;
            proxy_redirect      off;

            proxy_set_header   Host             $host;
            proxy_set_header   X-Real-IP        $remote_addr;
            proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        }
}

This tells nginx to listen to port 80, for incoming connections on any hostname, and forward it to our application.

Multiple Full-Screens in Java

Working with Java’s Full-Screen Exclusive Mode is a bit different than working with AWT, Swing or JavaFX. The tutorial describes how, and why, this works. The information, however, is a bit spread out. Also, it doesn’t mention how to work with multiple full screens at the same time. Not that it’s very different from working with only one screen, but it’s at least fun to try out.

The Frame

The first part of this exercise is to create a Frame that should be displayed. Since the OS is managing the video memory, we should guard against it. We could lose our drawing at any time, because the OS can simply reclaim the memory. The draw method looks a bit complex because of this, with its double loop structure. But on the positive side, we can just ignore any hint by the OS that the frame should be repainted.

package nl.ghyze.fullscreen;

import java.awt.Color;
import java.awt.Frame;
import java.awt.Graphics;
import java.awt.image.BufferStrategy;

public class TestFrame extends Frame {
    private final String id;

    public TestFrame(String id) {
        this.id = id;

        // ignore OS initiated paint events
        this.setIgnoreRepaint(true);
    }

    public void draw() {
        BufferStrategy strategy = this.getBufferStrategy();
        do {
            // The following loop ensures that the contents of the drawing buffer
            // are consistent in case the underlying surface was recreated
            do {
                // Get a new graphics context every time through the loop
                // to make sure the strategy is validated
                Graphics graphics = strategy.getDrawGraphics();

                int w = this.getWidth();
                int h = this.getHeight();

                // clear screen
                graphics.setColor(Color.black);
                graphics.fillRect(0,0,w,h);

                // draw screen
                graphics.setColor(Color.ORANGE);
                graphics.drawString("Screen: " + this.id, w / 2, h / 2);

                // Dispose the graphics
                graphics.dispose();

                // Repeat the rendering if the drawing buffer contents
                // were restored
            } while (strategy.contentsRestored());

            // Display the buffer
            strategy.show();

            // Repeat the rendering if the drawing buffer was lost
        } while (strategy.contentsLost());
    }
}

The ScreenFactory

This is just a simple utility class to figure out which screens are available, and what quality images we can show on these screens.

package nl.ghyze.fullscreen;

import java.awt.DisplayMode;
import java.awt.GraphicsDevice;
import java.awt.GraphicsEnvironment;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors;

public class ScreenFactory {

    private final GraphicsDevice[] graphicsDevices;

    /**
     * Constructor. Finds all available GraphicsDevices.
     */
    public ScreenFactory(){
        GraphicsEnvironment localGraphicsEnvironment = GraphicsEnvironment.getLocalGraphicsEnvironment();
        graphicsDevices = localGraphicsEnvironment.getScreenDevices();
    }

    /**
     * Get a list of the IDs of all available GraphicsDevices
     * @return the list of the IDs of all available GraphicsDevices
     */
    public List<String> getGraphicsDeviceIds(){
        return Arrays.stream(graphicsDevices).map(GraphicsDevice::getIDstring).collect(Collectors.toList());
    }

    /**
     * Get a single GraphicsDevice, by ID. Return an empty optional if none is found.
     * @param graphicsDeviceId the ID of the requested GraphicsDevice
     * @return an optional which contains the GraphicsDevice, if found.
     */
    public Optional<GraphicsDevice> getGraphicsDevice(String graphicsDeviceId){
        return Arrays.stream(graphicsDevices)
                .filter(graphicsDevice -> graphicsDevice.getIDstring().equals(graphicsDeviceId))
                .findAny();
    }

    /**
     * Get all available DisplayModes for the selected GraphicsDevice
     * @param graphicsDeviceId the ID of the GraphicsDevice
     * @return a list of DisplayModes
     */
    public List<DisplayMode> getDisplayModes(String graphicsDeviceId){
        GraphicsDevice gd = Arrays.stream(graphicsDevices)
                .filter(graphicsDevice -> graphicsDevice.getIDstring().equals(graphicsDeviceId))
                .findFirst()
                .orElseThrow(IllegalArgumentException::new);

        return Arrays.stream(gd.getDisplayModes()).collect(Collectors.toList());
    }

    /**
     * Get the best DisplayMode for the selected GraphicsDevice.
     * Best is defined here as the most pixels, highest bit-depth and highest refresh-rate.
     * @param graphicsDeviceId the ID of the GraphicsDevice
     * @return the best DisplayMode for this GraphicsDevice
     */
    public DisplayMode getBestDisplayMode(String graphicsDeviceId){
        List<DisplayMode> displayModes = getDisplayModes(graphicsDeviceId);
        DisplayMode best = null;
        for (DisplayMode displayMode : displayModes){
            if (best == null){
                best = displayMode;
            } else {
                if (isScreensizeBetterOrEqual(best, displayMode) ){
                    best = displayMode;
                } else if (isScreensizeBetterOrEqual(best, displayMode)
                        && isBitDepthBetterOrEqual(best, displayMode)){
                    best = displayMode;
                } else if (isScreensizeBetterOrEqual(best, displayMode)
                        && isBitDepthBetterOrEqual(best, displayMode)
                && isRefreshRateBetterOrEqual(best, displayMode)){
                    best = displayMode;
                }
            }
        }
        return best;
    }

    private boolean isScreensizeBetterOrEqual(DisplayMode current, DisplayMode potential){
        return potential.getHeight() * potential.getWidth() >= current.getHeight() * current.getWidth();
    }

    private boolean isBitDepthBetterOrEqual(DisplayMode current, DisplayMode potential){
        if (current.getBitDepth() == DisplayMode.BIT_DEPTH_MULTI) {
            return false;
        } else if (potential.getBitDepth()  == DisplayMode.BIT_DEPTH_MULTI){
            return true;
        }
        return potential.getBitDepth() >= current.getBitDepth();
    }

    private boolean isRefreshRateBetterOrEqual(DisplayMode current, DisplayMode potential){
        if (current.getRefreshRate() == DisplayMode.REFRESH_RATE_UNKNOWN) {
            return false;
        } else if (potential.getRefreshRate()  == DisplayMode.REFRESH_RATE_UNKNOWN){
            return true;
        }
        return potential.getRefreshRate() >= current.getRefreshRate();
    }
}

Bringing it together

For every screen that we have, we’re going to make a Frame. If the screen supports a Full Screen mode, we’re going to use it. Otherwise, we’re just going to use a maximized Frame. Once we’ve setup the screens, we’re going to loop over the frames, and draw each of them. We’ll do this in an infinite loop until we reach some stop condition. In this case, we’re going to stop after two seconds, but you can implement it in any way you’d like. I’ve found that if you don’t implement a stop condition, and just use an infinite loop, it can be quite challenging to actually stop the program. When the program should shut down, we’re going to reset the screens to normal and dispose of the Frames. Once the Frames are disposed, the program stops.

package nl.ghyze.fullscreen;

import java.awt.DisplayMode;
import java.awt.GraphicsDevice;
import java.util.ArrayList;
import java.util.List;

public class MultiFullScreen {
    private final List<TestFrame> frames = new ArrayList<>();

    private final long startTime = System.currentTimeMillis();
    private final ScreenFactory screenFactory = new ScreenFactory();

    public MultiFullScreen(){
        try {
            for (String graphicsDeviceId : screenFactory.getGraphicsDeviceIds()) {
                GraphicsDevice graphicsDevice = screenFactory.getGraphicsDevice(graphicsDeviceId).orElseThrow(IllegalStateException::new);
                DisplayMode best = screenFactory.getBestDisplayMode(graphicsDeviceId);

                TestFrame tf = new TestFrame(graphicsDeviceId);
                // remove borders, if supported. Not needed, but looks better.
                tf.setUndecorated(graphicsDevice.isFullScreenSupported());

                // first set fullscreen window, then set display mode
                graphicsDevice.setFullScreenWindow(tf);
                graphicsDevice.setDisplayMode(best);

                // can only be called after it has been set as a FullScreenWindow
                tf.createBufferStrategy(2);

                frames.add(tf);
            }
            run();
            shutDown();
        }catch (Exception e){
            e.printStackTrace();
        } finally {
            shutDown();
        }
    }

    private void shutDown() {
        // unset full screen windows
        for (String graphicsDeviceId : screenFactory.getGraphicsDeviceIds()) {
            try {
                GraphicsDevice graphicsDevice = screenFactory.getGraphicsDevice(graphicsDeviceId).orElseThrow(IllegalStateException::new);
                graphicsDevice.setFullScreenWindow(null);
            } catch (Exception e){
                e.printStackTrace();
            }
        }

        // Dispose frames, so the application can exit.
        for (TestFrame frame : frames){
            frame.dispose();
        }
    }

    public void run(){
        while(shouldRun()){
            for(TestFrame tf : frames){
                tf.draw();
            }
        }
    }

    private boolean shouldRun(){
        final long runningTime = System.currentTimeMillis() - startTime;
        return runningTime < 2000L;
    }

    public static void main(String[] args) {
        new MultiFullScreen();
    }
}

Jackson, JAXB, OpenAPI and Spring

Imagine that you have a service that enriches some data. The model for the data that needs to be enriched is generated from an XSD. The model for the enriching data (the wrapper) is generated from OpenAPI specs. Those two separate models need to be combined and sent out via Spring’s RestTemplate. The problem is that the generated models interfere with each other, and the combination of models doesn’t serialize correctly.

To solve the problem of interference, we’re going to transform the XSD-generated model to a generic JSON model (using the org.json library). That way we eliminate the XML annotations, and we can just send the OpenAPI model.

XML model to JSONObject

Our wrapper model accepts any Object as payload. We can set our XmlModel there, but we can also set a JSONObject. However, we need to transform our XML model to JSONObject. We can use Jackson for this, when we use the right AnnotationInspector. We’re not going to transform the model directly to JSONObject, but we use a String representation as an intermediate step. Once we have converted our XmlModel to JSONObject, we can set that as our payload.

private JSONObject toJSONObject(XmlModel xmlModel) {
  try {
    ObjectMapper objectMapper = new ObjectMapper();
    objectMapper.setAnnotationIntrospector(
    new JaxbAnnotationIntrospector(TypeFactory.defaultInstance()));
    String content = objectMapper.writeValueAsString(xmlModel);
    return new JSONObject(content);
  } catch (JsonProcessingException e) {
    // Shouldn't happen.
    log.severe("Error translating to JSONObject");
    throw new IllegalStateException(
      "Error translating to JSONObject");
  }
}

Configure RestTemplate

Now we have a model that can serialize, but our RestTemplate can’t handle it yet. Specifically, The ObjectMapper that the RestTemplate uses can’t handle the JSONObject. But we can configure our custom ObjectMapper and tell the RestTemplate to use that one. To serialize the JSONObject, we need to add the JsonOrgModule. And while we’re at it, we’re going to add the JavaTimeModule so we can serialize dates and times. We can’t set the ObjectMapper directly, we need to set it on a MessageConverter. Then we need to explicitly set our MessageConverter as the first to be sure that it’s going to be used.

@Bean
public RestTemplate myRestTemplate() {
  // Create and configurae
  ObjectMapper objectMapper = new ObjectMapper();
  objectMapper.registerModule(new JsonOrgModule());
  objectMapper.registerModule(new JavaTimeModule());

  // Configure MessageConverter
  MappingJackson2HttpMessageConverter converter = new MappingJackson2HttpMessageConverter();
  converter.setObjectMapper(objectMapper);

  // Configure RestTemplate
  RestTemplate restTemplate =
     // create the RestTemplate;
     restTemplate.getMessageConverters()
       .add(0, outputgeneratiebeheerMessageConverter);

  return restTemplate;
}

Maven dependencies

The artifact jackson-module-jaxb-annotations contains the JaxbAnnotationIntrospector that we used to serialize the XML model. The jackson-datatype-json-org contains the JsonOrgModule that we used to serialize the JsonObjects. These dependencies need to be added, in addition to jackson

<properties>
  <jackson.version>2.12.1</jackson.version>
</properties>

<dependencies>
  <dependency>
    <groupId>com.fasterxml.jackson.module</groupId>
    <artifactId>jackson-module-jaxb-annotations</artifactId>
    <version>${jackson.version}</version>
  </dependency>
  <dependency>
    <groupId>com.fasterxml.jackson.datatype</groupId>
    <artifactId>jackson-datatype-json-org</artifactId>
    <version>${jackson.version}</version>
  </dependency>
</dependencies>