OperationalError: database is locked

[simon04]

How to reproduce

While doing a POST operation on /ticket/5218, Trac issued an internal error.

(please provide additional details here)

Request parameters:

{'__FORM_TOKEN': u'474b119fd2d28e5a95ed2bca',
 'action': u'leave',
 'cnum': u'12',
 'comment': u'If I see it correctly, the only relation types missing are some from the Oxomoa scheme (`stop_area`, `stop_area_group`, `line`, \u2026).',
 'field_cc': u'',
 'field_component': u'Plugin validator',
 'field_description': u'Validator does not know type=site, associatedStreet, TMC, bridge and other either common used or approved relations-types. Instead you get a warning about an unknown relation-type.\r\n\r\nr3362 , o22065\r\n\r\ncu skyper',
 'field_keywords': u'unknown relation type',
 'field_priority': u'normal',
 'field_summary': u'Validator does not know many relation-types',
 'field_type': u'defect',
 'field_version': u'latest',
 'id': u'5218',
 'replyto': u'',
 'submit': u'Submit changes',
 'ts': u'2011-12-10 22:51:08.318444+00:00'}

User agent: Mozilla/5.0 (X11; Linux x86_64; rv:8.0.1) Gecko/20100101 Firefox/8.0.1

System Information

System information not available

Enabled Plugins

Plugin information not available

Python Traceback

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.2-py2.6.egg/trac/web/main.py", line 511, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.2-py2.6.egg/trac/web/main.py", line 237, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.2-py2.6.egg/trac/ticket/web_ui.py", line 169, in process_request
    return self._process_ticket_request(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.2-py2.6.egg/trac/ticket/web_ui.py", line 537, in _process_ticket_request
    valid = self._validate_ticket(req, ticket, not valid) and valid
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.2-py2.6.egg/trac/ticket/web_ui.py", line 1182, in _validate_ticket
    for field, message in manipulator.validate_ticket(req, ticket):
  File "build/bdist.linux-x86_64/egg/tracspamfilter/adapters.py", line 77, in validate_ticket
    FilterSystem(self.env).test(req, author, changes, ip)
  File "build/bdist.linux-x86_64/egg/tracspamfilter/api.py", line 180, in test
    score, ['%s (%d): %s' % r for r in reasons]).insert()
  File "build/bdist.linux-x86_64/egg/tracspamfilter/model.py", line 138, in insert
    '\n'.join(self.reasons)))
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.2-py2.6.egg/trac/db/util.py", line 65, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.2-py2.6.egg/trac/db/sqlite_backend.py", line 78, in execute
    result = PyFormatCursor.execute(self, *args)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.2-py2.6.egg/trac/db/sqlite_backend.py", line 56, in execute
    args or [])
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.2-py2.6.egg/trac/db/sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
OperationalError: database is locked

[skyper]

me too 2 days ago and today, but only for a short time. A reload brought it back.

[stoecker]

We need another server to fix this issue.

[Don-vip]

What is the current server hardware ?

[stoecker]

The problem is that the server is overloaded with the openstreetmap.de tile server and this causes trouble with JOSM web-interface.

A normal virtual server for e.g. 8 EUR a month should work fine, but we (JOSM) have no money and I tried once to raise funds for JOSM and wont do it again.

[simon04]

Has the migration to some open source hosting services like GitHub been considered?

[stoecker]

Replying to simon04:

Has the migration to some open source hosting services like GitHub been considered?

Hosting is not enough. We need server access, as there is too much stuff running behind the back, which normal hosting providers wont give us. There is no sense in dropping all this additional stuff.

[mor@…]

A normal virtual server for e.g. 8 EUR a month should work fine, but we (JOSM) have no money and I tried once to raise funds for JOSM and wont do it again.

I'm willing to spend 8 EUR / Month for that for one year (and maybe beyond but at least for one year). Send the details to mor - AT - sommerfuchs.info.

Even better would be a Flattr Button, than I would already have done so.

[stoecker]

@bastiK, framm:

I think you two should comment here as well.

[stoecker]

BTW: I already created a Flattr account, but never activated it on the JOSM pages.

[framm]

Dirk - were you not the one who changed my drastic access limits to openstreetmap.de tiles, claiming that allowing users to download more and faster would not harm performance? Have you already set those limits back to the old settings and are we still overloaded? - As far as I know, FOSSGIS is planning to buy a new server which could then take on the tile server role and that would free up resources. Otherwise, if that doesn't work, we can request a cheap virtual machine to be run by FOSSGIS for us. No need to set up our own funds; also it would be undesirable to have JOSM rely on servers operated by private individuals.

[Don-vip]

openstreetmap.de relies only on one server ? We have some unused servers, at OpenStreetMap France, which could be helpful for hosting JOSM :) If this idea pleases you, do you want I make an official request to OSM-FR board ? FYI we currently have 8 running servers (​http://wiki.openstreetmap.org/wiki/FR:Servers), and I think 2 more are being set up.

[bastiK]

Replying to framm:

As far as I know, FOSSGIS is planning to buy a new server which could then take on the tile server role and that would free up resources.

Sounds good, any estimate when this will happen?

Otherwise, if that doesn't work, we can request a cheap virtual machine to be run by FOSSGIS for us. No need to set up our own funds; also it would be undesirable to have JOSM rely on servers operated by private individuals.

Regarding donations, it would be nice if we could work more closely with FOSSGIS and they could do the "bookkeeping". E.g. Paypal and flattr buttons where donations go to a FOSSGIS bank account, but are only spent to support the JOSM project.

E.g. they could bill the running costs of the server, but ideally wouldn't turn it off when there are "insufficient funds". The remaining money would be available for other thinks, e.g. we could buy a CA certificate or pay someone for graphical design work (if needed).

[stoecker]

Replying to framm:

Dirk - were you not the one who changed my drastic access limits to openstreetmap.de tiles, claiming that allowing users to download more and faster would not harm performance?

Well, crippling openstreetmap.de for JOSM's sake is also not the right way. BTW I believe that the software on this system is broken. Either Apache, the tile handler or something in the system is wrong. That now JOSM page has some issues does not mean I regret to optimizations to tile.openstreetmap.de. The openstreetmap.de tiles are now much better usable :-)

I can always simply move JOSM to one of our company servers, but this also means I would be the only admin and this is not really a good way.

Have you already set those limits back to the old settings and are we still overloaded? - As far as I know, FOSSGIS is planning to buy a new server which could then take on the tile server role and that would free up resources. Otherwise, if that doesn't work, we can request a cheap virtual machine to be run by FOSSGIS for us. No need to set up our own funds; also it would be undesirable to have JOSM rely on servers operated by private individuals.

The data delivery load is still not the problem. Problems are the rendering load in one case and bugs in apache/mode_tile/whatever which cause hanging connections. Also the slow harddisk probably causes issues. I currently need to restart apache every second day to keep it working. Maybe updating the server to most recent distribution could fix most of the issues? It could also be that one of the Trac components is the cause of trouble.

But moving josm to an own server is probably about time, as not only tile.openstreetmap.de load increased, but also the load for JOSM service itself (especially database is much larger and more actively used).

@framm:
Would it be possible that FOSSGIS pays one of the 8EUR or 12EUR Hetzner VServers and we accept donations through FOSSGIS for this?

Replying to Don-vip:

openstreetmap.de relies only on one server?

No. The domains josm.openstreetmap.de and tile.openstreetmap.de are on one server. And both get more actively used and for both increases I'm one of the reasons. Now I face the resulting troubles :-)

If this idea pleases you, do you want I make an official request to OSM-FR board?

Would be fine. We would need a solution for at least the next 3 years. Required is linux root access to the server for at least two of us (bastiK, me). This means: If we change, we need a better solution than what we have.

There are no problems finding a solution, the task is to find a good one :-)

[stoecker]

To know what I'm talking about: On this 12,90EUR root server ​http://www.hetzner.de/en/hosting/produkte_vserver/vq12 I'm running

• a DNS server
• multiple web pages
• a mail server for about 20 domains
  • including spam filter
  • IMAP and POP access
  • web mail interface

without any trouble. So the same server would be good for josm for the next years. I believe the cheaper 8EUR one could be enough as well :-)

[Don-vip]

Replying to stoecker:

If this idea pleases you, do you want I make an official request to OSM-FR board?

Would be fine. We would need a solution for at least the next 3 years. Required is linux root access to the server for at least two of us (bastiK, me). This means: If we change, we need a better solution than what we have.

Request sent to OSM-FR, awaiting answers :)

[Don-vip]

Replying to stoecker:

If this idea pleases you, do you want I make an official request to OSM-FR board?

Would be fine. We would need a solution for at least the next 3 years. Required is linux root access to the server for at least two of us (bastiK, me). This means: If we change, we need a better solution than what we have.

Request sent to OSM-FR, awaiting answers :)

First answered received, they are highly positive for such a project :) It will be discussed further on next IRC board meeting (Monday).

They have however some practical questions:

• What are the services installed and the root priviliges needed for ? I already listed Apache, SVN, Trac, database, Java generation workshop (jdk, ant) and the need for a web proxy (ie Yahoo), but I'm not aware of everything :)
• What is the estimated bandwidth ?
• What is the current disk usage ?

[simon04]

Any news on this?

[Don-vip]

Replying to simon04:

Any news on this?

I misunderstood the board meeting date, the next meeting will in fact happen after the holidays :)

[stoecker]

FOSSGIS bought a new server and probably installs it in January. They also want to continue support of JOSM by offering server hosting. So very likely we can move to a VM on this server.

So I think in January there will be changes for sure and we will soon see in which direction we will go :-)

[Don-vip]

The OSM France board met yesterday evening. They're officially OK to welcome us on a private virtual server with root access. The physical server (HP DL360 G4) is currently being set up in Grenoble, France: ​http://wiki.openstreetmap.org/wiki/FR:Servers/Hardware

[Don-vip]

Did the change occur with FOSSGIS ? I don't know if the database was temporary locked or something, but my changeset r5046 does not show up on Trac :(

[rickmastfan67]

Replying to Don-vip:

Did the change occur with FOSSGIS ? I don't know if the database was temporary locked or something, but my changeset r5046 does not show up on Trac :(

It does look like it was built into JOSM still, as the "latest" JOSM is r5046.

[simon04]

Any new server in sight?

OperationalError occurred again on one of the previous days …

[stoecker]

Currently I'm lacking time. The new server is ready and base installation done, but the last two weekends I had no time (one week I was ill, the second simply had no interest to do anything for JOSM :-). Next week I'm going to hold two presentations, so this weekend probably there will be no time as well. I hope at 17.03.-18.03. the server move will start. :-)

[simon04]

Thanks for the update. :-)

I hope at 17.03.-18.03. the server move will start. :-)

So just in time for the FOSSGIS ;-)

[skyper]

Ticket #7152 has been marked as a duplicate of this ticket.

[skyper]

I had a look at Trac's Trac and found:
​http://trac.edgewall.org/search?q=!OperationalError%3A+database+is+locked

Hope it is really related to the hardware !

This page is nice, too: ​trac:/wiki/MostFrequentDuplicates

[Don-vip]

The database is locked again :(

EDIT: I have re-submitted my ticket comment and this time, it worked. Strange.

[skyper]

Replying to Don-vip:

The database is locked again :(

EDIT: I have re-submitted my ticket comment and this time, it worked. Strange.

It is often locked for a short time, sometimes a reload works right away.

[stoecker]

Replying to Don-vip:

The database is locked again :(

EDIT: I have re-submitted my ticket comment and this time, it worked. Strange.

The lock is a pretty normal thing. What is not normal are the long lock times and pending locks. These are caused by high load and Apache errors. I'm currently setting up the new server, but due to good weather conditions on weekend my computer time was a bit reduced :-)

[bastiK]

Hi Dirk, hope the good weather wasn't too distracting. I'm looking forward to seeing the new server in action! :)

[stoecker]

Moved to the new server and new apache and new python and new....

Hopefully this fixes this issue.

There still may be looking issues, but they should be very seldom.

[stoecker]

This issue seems still unfixed. I now start to investigate a bit deeper, no need to reopen!

It seems like this is caused by very expensive trac operations done by web-spiders (Baidu beeing the most aggressive). As JOSM Trac is very active, a full history of timeline or equal requests are VERY expensive.

The new server reduced this issue, but surely has not enough power to do these operations in short time. :-)

[skyper]

Replying to stoecker:

This issue seems still unfixed. I now start to investigate a bit deeper, no need to reopen!

+1

Do you need any specific information when it accures ?

[skyper]

Just happened again, 21:03 CEST.

---

How to reproduce

While doing a GET operation on /report/9, Trac issued an internal error.

(please provide additional details here)

Request parameters:

{'id': u'9'}

User agent: #USER_AGENT#

System Information

System information not available

Enabled Plugins

Plugin information not available

Python Traceback

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Trac-0.12.3-py2.7.egg/trac/web/main.py", line 522, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.7/dist-packages/Trac-0.12.3-py2.7.egg/trac/web/main.py", line 266, in dispatch
    req.session.save()
  File "/usr/local/lib/python2.7/dist-packages/Trac-0.12.3-py2.7.egg/trac/web/session.py", line 105, in save
    @self.env.with_transaction()
  File "/usr/local/lib/python2.7/dist-packages/Trac-0.12.3-py2.7.egg/trac/db/api.py", line 78, in transaction_wrapper
    ldb.commit()
OperationalError: database is locked

[skyper]

once more on the same request.

I also did not get any mail since 13:40 CEST but I should have as I got one some minutes ago sent at 13:42 CEST.

[stoecker]

I said:

no need to reopen!

I don't need additional info, as what you see is never the cause, but always the result.

As I already said database locks are pretty normal. What is not normal is e.g. that a chinese spider is requesting again and again one of the most resource needing links and this takes each time more than 2 minutes on the new machine.

This needs finetuning and there is no need for any user input. And very likely it will take weeks until I find the right parameters so this issue vanishs. The simple and fast solution would be to prevent spiders, but actually I think this is contra-productive.

[skyper]

Replying to stoecker:

I said:

no need to reopen!

Sorry.

I don't need additional info, as what you see is never the cause, but always the result.

[stoecker]

It is one of the problems you have when you have success. Everything gets complicated more and more and when you fixed a set of troubles there already is new trouble ready to step in. :-)

I reduced default maxdepth of entries for RSS pages from unlimited to 100. Maybe that reduces the load issue drastically.

[rickmastfan67]

Ticket #7667 has been marked as a duplicate of this ticket.