The Lost Update - Java, Spring and JPA -

i'm having issue @ work, tried months solve , it's driving me nuts.

the thing hard explain, involves particularities of domain i'm not allowed discuss, , can't copy-paste exact code. i'll try make myself clear can representative examples.

briefly speaking, system consists in root entity, let's call maindocument entity. around entity, there several entities orbiting. maindocument entity has state. let's call state "maindocumentstate".

public class maindocument {    @onetoone    @joincolumn(name = "document_state_id")    maindocumentstate state;    @version    long version = 0l; } 

there around 10 states available, on example focus on 2 of them. lets call them, readyforauthorization , authorized.

that's need know example.

about technologies using:

1. spring
2. gwt webapp
3. java 1.6
4. hibernate
5. jpa
6. oracle db.

about issue itself:

there section of system critical, , handles of incoming traffic. let's call section "the authorization section". on section, send information via soap ws provided our country's customs , border protection, authorize maindocument against customs.

the code looks this:

@transactional public void authorize(integer maindocid) {    maindocument maindocument = maindocumentservice.findbyid(maindocid);   // if document not found, exception thrown.   assert.istrue(maindocument.notautorized(), "the document       authorized");   // more bussiness logic validations happen here. validations not important topic discussed here. make sure document meets basic preconditions.    try {     transaction atransaction = transactionservice.newtransaction(); // creates transaction, entity stored in database keeps track of authorization service calls    try {     response response = wsauthroizationservice.sendauthorization(maindocument.getid(), maindocument.getauthorizationid()); // take account call can take between 2-4 minutes.      catch (exception e) {      atransaction.failed();      transactionservice.saveorupdate(atransaction);      throw e;     }     // behaviour same every error code.     if (response.getcode() != 0) {      atransaction.seterrorcode(resposne.getcode());      transactionservice.saveorupdate(atransaction);      throw authroizationerror("error on auth");     }     atransaction.completed();     maindocument.setauthorizationcode(0);     maindocument.authorize(); // change state "authorized"    } catch (exception e) {     maindocument.authorize(); // not change state because   authorizationcode != 0 or null.    } {     saveorupdate(maindocument);    }   } 

when lost update happen , how affects system:

1. maindocument id: 1@thread-1 tries authorize
2. the document not authorized, execution continues
3. goes through webservice , authorizes ok
4. transaction closes , commit happens.
5. while 1 commiting, maindocument 1@thread-2 comes in, , tries to auth.
6. 1 not persisted yet, thread-2 tries auth.
7. thread-2 rejected ws response "the document 1 authorized".
8. thread-2 tries commit.
9. thread-1 commits first document 1, thread-2 commits in second place.

maindocument id:1 persisted state readyforauthorization, while correct state should authorized.

the complexity arises because it's impossible reproduce. happens in production , if try flood server hundreds of calls, can't same behavior.

implemented solutions:

1. thread barrier, if 2 threads same maindocument id try authorize, last enter rejected. it's implemented aspect, order 100, it's executed after @transactional commit. tested , checked on stacktrace transaction commits before aspect intercepts , removes thread barrier.
2. @version, works on other sections of system, raising optimisticlockexception when 1 commit tries override commit older transaction. in case optimisticlockexception not being raised.
3. "transaction" persisted @transactional(propagation = requires_new) it's independent main transaction , it's commited correctly. transactions it's clear lost update issue, because can see completed transaction success message, , maindocument persisted different state, no errors showing on server.log.
4. using imperva securesphere can audit updates on specific table. can see first transaction commiting correct state , second transaction overwriting first.

i grateful if concurrency , transaction managing experience can give me useful tips on how debug or reproduce issue, or @ least implement solutions mitigate damages.

to clear, there more 1000 request per hour , 99.99% of these requests end correctly. total number of cases problem present 20 per month.

added 09-13-17:

the saveorupdate method using , if needed:

   * "http://blog.xebia.com/2009/03/23/jpa-implementation-patterns-saving-detached-entities/" >jpa    * implementation patterns: saving (detached) entities</a>    *     * @param entity    */   protected e saveorupdate(e entity) {     if (entity.getid() == null) {       getjpatemplate().persist(entity);       return entity;     }     if (!getjpatemplate().getentitymanager().contains(entity)) {       return merge(entity);     }     return entity;   } 

the main problem concurrency. way code looks now, it's trying check if entity authorized, when should check if authorized or in process of being authorized.

it leads important question: how check if entity being manipulated across system?

i've faced situations similar, including scenarios code running in clusters. best working solution found use form o database lock.

the @version should , quick solution, stated it's not working properly. stated can audit database using tool, interesting check how version field behaving in case.

with no @version, try "hardcore" pessimistic database lock. proposed solution not only, or best one.

1 - create new table. table store ids of documents being processed. pk should document id, or else ensures same document won't have duplicates in table.

2 - in code, before retrieving entity, check if id in table created in step 1. if it's not, go ahead. if is, assume being processed , nothing.

3 - in code, right after retrieving entity, must insert id in table created in step 1.
if document not being authorized, insert successful , process continues.
if chance, 2 requests being executed @ same time, 1 of requests constraint violation exception (or similar). code should assume document being authorized.
important: insertion must executed in new transaction. spring bean used persist id in new table should have it's methods marked @transaction(propagation = propagation.requires_new).

4 - after webservice called , response processed, remove id table created in step 1. should executed in separated transaction.
consider doing in block, because if other runtime error occurs, document id should removed table.

how debug:

• run app in local environment, , put breakpoint right after entity retrieved , before insertion in new table. if want debug current code, put breakpoint right after assert statement.

• open 2 different browsers in dev machine, , perform use case triggers code. can ask team member perform machine.

• you should see ide showing code being executed @ breakpoint twice. after let both executions run, 1 after another, , enjoy show. scenario should reproduced.

• basically emulates 2 simultaneous requests.

considerations:

• i choose use database table because solution work if app deployed in cluster environment (multiple app server instances).
• if there single instance running, try using object shared across requests, if in future need scale app using clusters, solution won't work. you'll have deal thread safety.
• you try using database locking, have careful not lock table/row long. also, jpa doesn't have specific operation perform locks on tables/rows (at least could't find one), have deal native sql.