Modify

Opened 4 years ago

Last modified 4 years ago

#21164 new enhancement

Reading folder with image sequence over network is slow

Reported by: cyton Owned by: team
Priority: normal Milestone:
Component: Core image mapping Version: tested
Keywords: template_report network exif performance cache Cc:

Description

What steps will reproduce the problem?

  1. Drag folder containing Images (in this case 11488 at 27G) from nautilus into josm.
  2. wait a long time, several minutes
  3. have images ready as usual

What is the expected result?

The wait time should be far smaller, reading the same sequence from a local hard drive is comparatively extremely fast

What happens instead?

JOSM goes through three steps which each take at least a minute;

  1. "Looking for image files\nopening 1 file..." this takes about 2 minutes. iftop shows network activity to and from my NAS at about 500kb, htop shows 0.0 cpu usage for the josm jar
  1. "Starting directory scan" this takes about 3 minutes. iftop shows network activity to and from my NAS at about 550kb, htop shows 0.0 cpu usage for the josm jar
  1. "Reading G0123456.JPG..." and so on, this takes about 16 minutes. iftop shows network activity from my NAS at about 60Mb, and to it at about 900kb. htop shows 0.0 cpu usage for the josm jar

For comparison, reading the same files from a local hard drive takes about 6 seconds, not 21 minutes.

Please provide any additional information below. Attach a screenshot if possible.

My relevant home network consists of two PCs, two USB 2.5G ethernet dongles and two 10G switches.
iperf shows a speed between the two of 2.5Gigabit, that should not be a problem.

Copying the folder from my NAS to my PC:
rsync -Pvhtr --info=progress2 --info=name0 --stats --progress source_on_NAS ~/destination
Takes about 19 minutes.
With a reported speed of about 23.5MB/s, iftop shows a transfer rate of about 200Mb during this.

Not surprisingly, this is the same amount of time it took JOSM for the above described step 3. of reading the image files EXIF information.

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2021-06-02 22:03:39 +0200 (Wed, 02 Jun 2021)
Revision:17919
Build-Date:2021-06-02 20:11:30
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (17919 en) Linux Ubuntu 20.04.2 LTS
Memory Usage: 1368 MB / 8022 MB (683 MB allocated, but free)
Java version: 11.0.11+9-Ubuntu-0ubuntu2.20.04, Ubuntu, OpenJDK 64-Bit Server VM
Look and Feel: com.sun.java.swing.plaf.gtk.GTKLookAndFeel
Screen: :0.0 1400×1050 (scaling 1.00×1.00) :0.1 2560×1440 (scaling 1.00×1.00) :0.2 2560×1440 (scaling 1.00×1.00)
Maximum Screen Size: 2560×1440
Best cursor sizes: 16×16→16×16, 32×32→32×32
Environment variable LANG: en_US.UTF-8
System property file.encoding: UTF-8
System property sun.jnu.encoding: UTF-8
Locale info: en_US
Numbers with default locale: 1234567890 -> 1234567890
Desktop environment: ubuntu:GNOME
Java package: openjdk-11-jre:amd64-11.0.11+9-0ubuntu2~20.04
Java ATK Wrapper package: libatk-wrapper-java:all-0.37.1-1
libcommons-logging-java: libcommons-logging-java:all-1.2-2
fonts-noto: fonts-noto:all-20200323-1build1~ubuntu20.04.1

Plugins:
+ FastDraw (35640)
+ ImportImagePlugin (35567)
+ KartaView (374)
+ Lanes (${version.entry.commit.revision})
+ Mapillary (2.0.0-alpha.26-dirty)
+ PicLayer (1.0.1)
+ PolygonCutOut (v0.7)
+ RoadSigns (35640)
+ apache-commons (35524)
+ apache-http (35589)
+ buildings_tools (35756)
+ editgpx (35562)
+ ejml (35458)
+ geotools (35458)
+ jaxb (35543)
+ jna (35662)
+ jogl (1.2.3)
+ jts (35458)
+ kendzi3d (1.0.205)
+ kendzi3d-resources (0.0.2)
+ log4j (35458)
+ measurement (35640)
+ photo_geotagging (35738)
+ photoadjust (35770)
+ print (35640)
+ tageditor (35640)
+ turnlanes-tagging (288)
+ turnrestrictions (35640)
+ utilsplugin2 (35691)

Tagging presets:
+ https://github.com/kendzi/Simple3dBuildingsPreset/releases/download/0.9_2018-05-08/s3db-preset.zip
+ https://josm.openstreetmap.de/josmfile?page=Presets/BuildingPreset&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/NewParkingFeatures&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Presets/ParkingLanes&zip=1

Map paint styles:
- https://josm.openstreetmap.de/josmfile?page=Styles/TreeCrownDiameter&zip=1
+ https://josm.openstreetmap.de/josmfile?page=Styles/iD&zip=1
- ${HOME}/josmBigNodes.mapcss
- https://josm.openstreetmap.de/josmfile?page=Styles/LessObtrusiveNodes&zip=1
- ${HOME}/_josmCyclosmTest.mapcss
- ${HOME}/_josmCyclosmTest_width.mapcss
- ${HOME}/_josmCyclosmTest_casing.mapcss
- ${HOME}/_josmBigNodes.mapcss

Last errors/warnings:
- 00004.154 W: Failed to load Mappaint styles from '${HOME}/josmBigNodes.mapcss'. Exception was: java.nio.file.NoSuchFileException: ${HOME}/josmBigNodes.mapcss
- 00004.157 E: java.nio.file.NoSuchFileException: ${HOME}/josmBigNodes.mapcss
- 00005.221 W: Cannot lock cache directory. Will not use disk cache
- 00008.095 W: Cannot start IPv4 remotecontrol server on port 8111: Address already in use (Bind failed)
- 00008.095 W: Cannot start IPv6 remotecontrol server on port 8111: Address already in use (Bind failed)
- 01615.173 E: Failed to locate image 'preference.png'

Attachments (0)

Change History (4)

comment:1 by cyton, 4 years ago

I would suggest a cache of the metadata read from EXIF, this would also speed up reading a lot of images stored on a local hard drive.

Or, if possible, only read the exif header from the image, in a way that does not need the whole image.

Maybe with a piece of software running on the NAS which JOSM contacts for only the EXIF information or to create the metadata cache, to speed this up too for the first read of the images.

comment:2 by skyper, 4 years ago

Component: CoreCore image mapping

comment:3 by skyper, 4 years ago

Keywords: network exif added

comment:4 by Don-vip, 4 years ago

Keywords: performance cache added

Modify Ticket

Change Properties
Set your email in Preferences
Action
as new The owner will remain team.
as The resolution will be set. Next status will be 'closed'.
to The owner will be changed from team to the specified user.
Next status will be 'needinfo'. The owner will be changed from team to cyton.
as duplicate The resolution will be set to duplicate. Next status will be 'closed'. The specified ticket will be cross-referenced with this ticket.
The owner will be changed from team to anonymous. Next status will be 'assigned'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.