Browsed by
Tag: howto

Snapshotting PostgreSQL databases with templates using Spring JDBC

Snapshotting PostgreSQL databases with templates using Spring JDBC

postgresql logo Nowadays with the rise of more and more complex systems software testing is more important then ever. This not only includes unit testing but also integration-, end to end- and acceptance tests. For some of these test scenarios it is important to have a defined state of your database which could be reproduced in example between different test suites. To achieve this goal there are different options. It is possible to simply rebuild a docker container in which the database runs and migrate to the actual state with a tool like flyway (here I described how to do that with a Java EE application) but this may take some time and seems only suitable between independent test runs. Another option would be to delete all table data (truncate) and repopulate all the needed test data with appropriate SQL Scripts. This should be the fastest option but it has the disadvantage that the SQL scripts has to be maintained manually and you could only reset a state that is kept in this SQL scripts.
When I was faced with this task I recently used a nice feature of the PostgreSQL database for a snapshot like behaviour. Postgres comes with a build-in template mechanism which is used when a new database is created. For this a freshly installed Postgres instance comes with a default database named ‘postgres’ and two templates named ‘template0’ and ‘template1’. The idea is that you can modify template1 as you like to suite your needs for creating new empty databases and let template0 as it is to still have a plain default. You can read more about this in the Postgres documentation. Additional to that it is possible to create your own custom templates which can be copies of actual databases. By doing that you can effectively “roll back” at any time later. In my case this was really fast (your mileage may vary depending on the size of your database), had a minimum of code to be written and comes with the benefit that it is possible to make the template at runtime when the tester has created a state he would like to reproduce.

I chose to do all the needed operations with Spring JDBC as it is very convenient because the use of the JdbcTemplate handles all the connecting/disconnecting and error handling for you. All you need to do is to pass it a datasource and use standard SQL statements as query parameters for the template. To learn more about it you can take a look at the excellent Spring documentation about data access with JDBC but I guess you will be able to follow my code snippets here as well.

@Component
public class DefaultDataSource extends DriverManagerDataSource {

  @PostConstruct
  public void init() {
    this.setDriverClassName("org.postgresql.Driver");
    this.setUrl("jdbc:postgresql://localhost:5432/postgres");
    this.setUsername("user");
    this.setPassword("password");
  }
}

This class defines a datasource for a standard Postgres db running on your local machine at the standard port. The @Component annotation is used by Spring  to identify it as a Spring bean which makes it injectable via the @Autowired annotation while the @PostConstruct annotation gets the init method called after construction of the object.

Read More Read More

Removing Excel rows with POI

Removing Excel rows with POI

https://poi.apache.org
https://poi.apache.org

When generating Excel Sheets it is sometimes necessary to remove rows which match a given criteria afterwards. The first idea for this would be to use removeRow(Row row) but this has the problem that it deletes all the row contents and leaves the empty row in your sheet which is probably not what you want. To remove the rows as a whole the shiftRows(int startRow, int endRow, int n) is needed.
Lets say we have a xls file with column A being the title which is always filled but we want to remove every row in which the data column B is not filled. For this case the following snippet would be appropriate

Read More Read More

eclipse workspace in use

eclipse workspace in use

eclipse logo

As I’m a huge fan of IntelliJ IDEA I don’t use eclipse very often during the last years but I have a dedicated project where I need to regularly use for creating deployments and connect with some remote test servers. So I have to deal with eclipse and its problems, too. A recurring problem for me with eclipse is that it stumbles upon its own not cleaned up metadata which causes problems that are not obvious to solve at first sight. Most of them are gladly quick to fix if you know what you have to do.

A common problem of that sort is when, after selecting the workspace, eclipse fires up a popup with the message “The default workspace … is in use or cannot be created. Please choose a different one”. Most of the time the workspace is not in use but the problem can be caused by an unexpected closing of eclipse which prevents it from cleaning up its metadata correctly. The solution for this is easy. Just navigate to the workspace folder and delete the .lock file in the hidden .metadata directory..

If you are using a mac like me the easiest way is to do it on the shell because the finder by default doesn’t show hidden folders.

cd ~/yourWorkspaceDirectory/.metadata
rm .lock 

Now eclipse should start with your workspace as expected.

Using CellUtil with POI to avoid too many cell styles in Excel

Using CellUtil with POI to avoid too many cell styles in Excel

https://poi.apache.org
https://poi.apache.org

When you create Excel documents with the popular Apache POI library it is possible that you encounter a problem where Excel tells you there are “Too many different cell formats”. A possible source for this is that you probably have a loop structure in which you define a new cell style for every cell you create.
poi-cellstyle-loop

Excel is limited in the amount of cell styles it can handle in a single document (4000 per document). Therefore the above code would break. The easy fix would be to define the cell style outside of your loop and reuse it.
But sometimes you want to create different cell styles depending on a line number for example. In this case POI comes with a handy utility class named CellUtil that allows you to modify existing cell styles without creating a new one. Internally it has to create a new cell style and saves it in a map. If you make the same modification at any other place in your code again it will iterate that map, find the previously created style and reuses it. Therefore the same style is only created once. The usage is quite simple and would adapt to the sample above as follows

poi cellUtil

The code above will add a border to the bottom of every second cell created but because it is every time the same modification CellUtil will only create one additional style and apply this every time again.

My git svn workflow

My git svn workflow

Git-Logo-2Color
Git Logo by Jason Long /CC BY 3.0

Using a remote subversion repository with git locally requires some adoptions to the local git usage compared to working in a pure git environment. The reason for this is that the remote subversion don’t know about many of the concepts the local git supports (local branches or multiple local commits before checkin for example). Because of this it is essential to make it look as close to a normal svn checkin as possible when using git svn.
As I now work since quite a while with such a setup I defined a workflow for me that don’t causes any problems with the svn and gives me many of the benefits I have with git as there are the possibility to work with the version history offline, making local branches to develop new features or making multiple commits when sub tasks are done.

Before starting to develop a new feature I need to make sure everything is up to date. In contrast to a git remote repository no pull option is available here but I have to rebase the svn repository. I call the following from the master branch:

git svn rebase

This gets any new code changes from the svn and rebases all local work done by me which is not checked in yet on top of it. This is essential because svn history goes in a straight line and would not know what to do if you push a local commit that is dated before the actual newest check in.
After having done that I can create a local feature branch using

git checkout -b newFeature

Here everything works as normal. You can make commits, branch out further, go back to a former version and so on. When you want to get the newest code from the remote repository merged into your branch I prefer pulling it in the master first and rebasing it from there in my featureBranch (it is important not to merge it)

git checkout master

git svn rebase

git checkout featureBranch

git rebase master

When the new feature is ready comes the essential part where the git svn mix breaks sometimes if you don’t do it right. As I mentioned it is essential to make the commit as equal to a svn commit as possible. When using svn you don’t have the possibility to make local commits you just push everything at once to the server where it gets a version id and thats it. To achieve that behaviour with git you change back to the master, checkout the newest code changes and merge it back to the feature branch as explained above. After that we can merge the feature branch back to master:

git checkout master

git merge featureBranch

When you look at the version history now you will see that every commit done in the feature branch is now at the top of the version history. This is because we rebased the master in the feature branch and not merged it. This gives us now the possibility to pack all our local commits to one commit which will then look like the normal subversion workflow to the repository. If we had done 5 commits in our local feature branch for example we achieve this with

git rebase --interactive master~5

This will guide you through two steps. In the first you have to select which of the five commits you want to squash together. Here you should squash all subsequent commits into the first. After that a new dialogue will open which needs you to define the new commit message. Here you see all commit messages you entered in the single commits and can copy the contents to a new single message. When you are done and saved the new message git will rebase everything to one commit which lies on top in the version history. The only thing left to do is push it to svn. Once again the usual git push won’t work here. Instead we call

git svn dcommit

That is all. Everything should work fine and your new feature is checked in to subversion. You can now delete the feature branch or keep to track back the single steps you made to develop the feature.

 

Update:

Just to clarify, the idea is not to work weeks long on a feature and try to integrate it with a big bang all at once. This was just meant for small features that require just a few hours or maybe a day of work. If you implement a larger feature you obviously should commit smaller portions of your code changes to svn to make your progress clear and reproducible and avoid large merge conflicts with other team mates changes. In the beginning I tried to dcommit all my single commits to svn which worked to a point but had one nearly fatal break. Since then I try to avoid that and use the described workflow.