Opened 4 years ago
Last modified 4 years ago
#21164 new enhancement
Reading folder with image sequence over network is slow
Reported by: | cyton | Owned by: | team |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | Core image mapping | Version: | tested |
Keywords: | template_report network exif performance cache | Cc: |
Description
What steps will reproduce the problem?
- Drag folder containing Images (in this case 11488 at 27G) from nautilus into josm.
- wait a long time, several minutes
- have images ready as usual
What is the expected result?
The wait time should be far smaller, reading the same sequence from a local hard drive is comparatively extremely fast
What happens instead?
JOSM goes through three steps which each take at least a minute;
- "Looking for image files\nopening 1 file..." this takes about 2 minutes.
iftop
shows network activity to and from my NAS at about 500kb,htop
shows 0.0 cpu usage for the josm jar
- "Starting directory scan" this takes about 3 minutes.
iftop
shows network activity to and from my NAS at about 550kb,htop
shows 0.0 cpu usage for the josm jar
- "Reading G0123456.JPG..." and so on, this takes about 16 minutes.
iftop
shows network activity from my NAS at about 60Mb, and to it at about 900kb.htop
shows 0.0 cpu usage for the josm jar
For comparison, reading the same files from a local hard drive takes about 6 seconds, not 21 minutes.
Please provide any additional information below. Attach a screenshot if possible.
My relevant home network consists of two PCs, two USB 2.5G ethernet dongles and two 10G switches.
iperf
shows a speed between the two of 2.5Gigabit, that should not be a problem.
Copying the folder from my NAS to my PC:
rsync -Pvhtr --info=progress2 --info=name0 --stats --progress source_on_NAS ~/destination
Takes about 19 minutes.
With a reported speed of about 23.5MB/s, iftop
shows a transfer rate of about 200Mb during this.
Not surprisingly, this is the same amount of time it took JOSM for the above described step 3. of reading the image files EXIF information.
Relative:URL: ^/trunk Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b Last:Changed Date: 2021-06-02 22:03:39 +0200 (Wed, 02 Jun 2021) Revision:17919 Build-Date:2021-06-02 20:11:30 URL:https://josm.openstreetmap.de/svn/trunk Identification: JOSM/1.5 (17919 en) Linux Ubuntu 20.04.2 LTS Memory Usage: 1368 MB / 8022 MB (683 MB allocated, but free) Java version: 11.0.11+9-Ubuntu-0ubuntu2.20.04, Ubuntu, OpenJDK 64-Bit Server VM Look and Feel: com.sun.java.swing.plaf.gtk.GTKLookAndFeel Screen: :0.0 1400×1050 (scaling 1.00×1.00) :0.1 2560×1440 (scaling 1.00×1.00) :0.2 2560×1440 (scaling 1.00×1.00) Maximum Screen Size: 2560×1440 Best cursor sizes: 16×16→16×16, 32×32→32×32 Environment variable LANG: en_US.UTF-8 System property file.encoding: UTF-8 System property sun.jnu.encoding: UTF-8 Locale info: en_US Numbers with default locale: 1234567890 -> 1234567890 Desktop environment: ubuntu:GNOME Java package: openjdk-11-jre:amd64-11.0.11+9-0ubuntu2~20.04 Java ATK Wrapper package: libatk-wrapper-java:all-0.37.1-1 libcommons-logging-java: libcommons-logging-java:all-1.2-2 fonts-noto: fonts-noto:all-20200323-1build1~ubuntu20.04.1 Plugins: + FastDraw (35640) + ImportImagePlugin (35567) + KartaView (374) + Lanes (${version.entry.commit.revision}) + Mapillary (2.0.0-alpha.26-dirty) + PicLayer (1.0.1) + PolygonCutOut (v0.7) + RoadSigns (35640) + apache-commons (35524) + apache-http (35589) + buildings_tools (35756) + editgpx (35562) + ejml (35458) + geotools (35458) + jaxb (35543) + jna (35662) + jogl (1.2.3) + jts (35458) + kendzi3d (1.0.205) + kendzi3d-resources (0.0.2) + log4j (35458) + measurement (35640) + photo_geotagging (35738) + photoadjust (35770) + print (35640) + tageditor (35640) + turnlanes-tagging (288) + turnrestrictions (35640) + utilsplugin2 (35691) Tagging presets: + https://github.com/kendzi/Simple3dBuildingsPreset/releases/download/0.9_2018-05-08/s3db-preset.zip + https://josm.openstreetmap.de/josmfile?page=Presets/BuildingPreset&zip=1 + https://josm.openstreetmap.de/josmfile?page=Presets/NewParkingFeatures&zip=1 + https://josm.openstreetmap.de/josmfile?page=Presets/ParkingLanes&zip=1 Map paint styles: - https://josm.openstreetmap.de/josmfile?page=Styles/TreeCrownDiameter&zip=1 + https://josm.openstreetmap.de/josmfile?page=Styles/iD&zip=1 - ${HOME}/josmBigNodes.mapcss - https://josm.openstreetmap.de/josmfile?page=Styles/LessObtrusiveNodes&zip=1 - ${HOME}/_josmCyclosmTest.mapcss - ${HOME}/_josmCyclosmTest_width.mapcss - ${HOME}/_josmCyclosmTest_casing.mapcss - ${HOME}/_josmBigNodes.mapcss Last errors/warnings: - 00004.154 W: Failed to load Mappaint styles from '${HOME}/josmBigNodes.mapcss'. Exception was: java.nio.file.NoSuchFileException: ${HOME}/josmBigNodes.mapcss - 00004.157 E: java.nio.file.NoSuchFileException: ${HOME}/josmBigNodes.mapcss - 00005.221 W: Cannot lock cache directory. Will not use disk cache - 00008.095 W: Cannot start IPv4 remotecontrol server on port 8111: Address already in use (Bind failed) - 00008.095 W: Cannot start IPv6 remotecontrol server on port 8111: Address already in use (Bind failed) - 01615.173 E: Failed to locate image 'preference.png'
Attachments (0)
Change History (4)
comment:1 by , 4 years ago
comment:2 by , 4 years ago
Component: | Core → Core image mapping |
---|
comment:3 by , 4 years ago
Keywords: | network exif added |
---|
comment:4 by , 4 years ago
Keywords: | performance cache added |
---|
I would suggest a cache of the metadata read from EXIF, this would also speed up reading a lot of images stored on a local hard drive.
Or, if possible, only read the exif header from the image, in a way that does not need the whole image.
Maybe with a piece of software running on the NAS which JOSM contacts for only the EXIF information or to create the metadata cache, to speed this up too for the first read of the images.