Garmin FIT import

[pschonmann]

Hi,
New garmin devices for bikes, like Edge series arent exporting tracks to GPX, but into FIT. Should be fine import that file directly without converting it in gpsbabel or other utility.

[skyper]

Is it not possible to change to export gpx ?

What is the advantage of FIT ?

[pschonmann]

No option to export/output file to GPX in device.
FIT file has less size than gpx even i record 1s and record HR, CADENCE from ant. Maybe this is some Garmin magic.

I dont know any advantage from gpx, but would be nice have direct import rather do some magic with convert tools thats hard to use for begginers.

[stoecker]

Nothing for the core.

[Don-vip]

This SDK should help: ​http://www.thisisant.com/resources/fit

[Don-vip]

Also, you should attach a .fit sample file if you want this ticket resolved some day, because developers do not have all devices released by Garmin :D

[mvestola]

Example fit file produced by Garmin Edge 800 during cycling

[mvestola]

Example fit file converted to GPX via Garmin Connect

[mvestola]

I would really need this feature too. I have Garmin Edge 800 which I use in my mountain bike. Edge 800 always outputs the GPS trail using the fit file format. The device has a very precise GPS unit and thus is an excellent device for OSM mapping. I know this is a very popular device with bikers.

However, it is a shame that I can't directly import the GPS trails from my Edge 800 to JOSM because the fit format is not supported by JOSM. But now I need to first convert the fit file to gpx via Garmin Connect website (because the linux version I use in my computer does not have the latest version of gpsbabel which has the fit file support and I don't like building the software from the git sources). I honestly must say that this limits my OSM mapping due to the time-consuming converting between fit and gpx formats.

I attached an example fit file I recorded with my Edge 800. I also attached the same GPS trail in GPX format. I converted the fit file to GPX with garmin connect website. I'm not 100% sure that the files have exactly the same information since the fit file is binary encoded but I assume that the Garmin connect produces the same GPS trail for the GPX file.

[simon04]

I obtained a working implementation of a FIT importer based on fit.jar from ​http://www.thisisant.com/resources/fit. I'm unsure whether the "Flexible and Interoperable Data Transfer (FIT) Protocol License" is compatible with GPL.

[bastiK]

Replying to simon04:

I obtained a working implementation of a FIT importer based on fit.jar from ​http://www.thisisant.com/resources/fit. I'm unsure whether the "Flexible and Interoperable Data Transfer (FIT) Protocol License" is compatible with GPL.

This license is definitely not compatible with GPL. What we could do:

• find a Java library for FIT under GPL
• port the ​GPSBabel code to Java
• work around the GPL restrictions, e.g.
  • Create a stand alone application (jar file) that can convert FIT to GPX. Make it available on the JOSM website.
  • Let the JOSM-Plugin download this application to the home folder, the first time it is needed. Do not link the jar file, but call it as an external application (something like new ProcessBuilder("java","~/.josm/fit2gpx.jar",fit_filename).start();)

[MaksVasilev]

Standalone fit2gpx: ​https://github.com/MaksVasilev/fit2gpx
binary jar: ​https://github.com/MaksVasilev/fit2gpx/blob/master/out/jar/fit2gpx.jar?raw=true

[MaksVasilev]

Open source (GPL2) software Golden Cheetah have native support FIT format:

​https://github.com/GoldenCheetah/GoldenCheetah/blob/master/src/FileIO/FitRideFile.cpp

GPX format a deprecated, and may be discontinued.

[yourealwaysbe]

I recently implemented a Java FIT to GPX conversion by porting the GPSBabel code instead of using the ANT SDK. It's not widely tested yet, but might be useful. I used the GPL license since that's what GPSBabel uses, and borrowed some tests from MaksVasilev's fit2gpx.

​https://gitlab.com/Hague/fitgpxconverter

[richlv]

Got an Edge device as a gift, and have a trace for editing, but it's in the fit format...
Any chance to get them supported by JOSM in near future?

[taylor.smock]

It looks like they have documented the protocol here: ​https://developer.garmin.com/fit/protocol/

There isn't a click-through license, so I might be able to do an independent implementation.

[richlv]

That would be marvelous. Would there be any benefit from test files?
Definitely happy to help with testing as much as I could.

[taylor.smock]

Would there be any benefit from test files?

Yes, if you don't mind them (a) being licensed in a compatible manner (GPLv2+, CC0, CC-BY, CC-BY-SA) and (b) in JOSM source code.

With that said, I don't know when I'll have the bandwidth to do this.

[richlv]

Sounds great - attaching two fit files that can be used for testing.
Absolutely understandable that this depends on time availability, will keep an eye on this ticket and try any new development whenever it arrives :)

[richlv]

FIT testcase 1, Edge 840 produced

[richlv]

FIT testcase 2, Edge 840 produced

[taylor.smock]

Sounds great - attaching two fit files that can be used for testing.

You attached gpx files, probably converted from fit files.

[richlv]

FIT testcase 1, Edge 840 produced

[richlv]

FIT testcase 2, Edge 840 produced

[richlv]

Ouch, such a fail. Sorry about that.
Uploaded FITs now. Looks like I cannot delete the GPX files - if somebody can do so, please go ahead.

[taylor.smock]

Looks like I cannot delete the GPX files

I'm not worried about that. In fact, the gpx files might be useful as a sanity check.

[taylor.smock]

In 36151/osm:

Fix #8647: Add Garmin FIT parsing (as a plugin)

Documentation/files used for implementation:

• ​https://developer.garmin.com/fit/protocol/ (no click-through license required at time of writing)
• FIT files attached to https://josm.openstreetmap.de/ticket/8647 along with decoded gpx files from gpsbabel (files provided by richlv)

[taylor.smock]

In 36152/osm:

See #8647 (dist): Add Garmin FIT parsing (as a plugin)

[taylor.smock]

Notes on the plugin:

• Requires Java 21 (I probably could have gone with Java 19, but that isn't an "LTS" release)
• Supports drag-n-drop
• Does not currently support remote URLs
• All decoded gps points are part of the same track

[richlv]

Wow, that's great, thank you so much :)
This finally got me to upgrade Java... but now it doesn't start, as a proxy apparently was configured that cannot be found now ("Creating of proxy failed"). This completely stop the startup.
Will test the new plugin as soon as I can get around this.

[richlv]

There seems to be a bit nasty bug in OpenWebStart - if it's set to "system settings" for proxy, system is set to a pac-file, and the host cannot be resolved, startup completely fails.
Found the settings app, manually changed the proxy setting for now.

Got stuck on Java version again - latest from ​https://openwebstart.com/download/ installed 17.0.7 Azul Systems, and the new plugin doesn't like it (as mentioned, requires 21).
Poking around the settings app, cannot find a way to get 21 there.

[richlv]

Downloaded OpenJDK21, added it in OpenWebStart settings, removed the Azul one.
On the next startup it downloads Azul again and uses that.
Perhaps caused by josm.jnlp having this?

<java version="1.8+" vendor="Azul"

[taylor.smock]

That might be the case. Azul does have a Java 21 JDK. See ​https://www.azul.com/downloads/?version=java-21-lts&package=jdk-fx#zulu .

[richlv]

Thanks, not sure what to do.
Stick with OpenJDK? If so, how to make OWS work with it?
Stick with Azul? If so, why - and why doesn't jnlp/OWS load 21?

[taylor.smock]

Stick with Azul? If so, why - and why doesn't jnlp/OWS load 21?

OWS needs to have the jvms.json updated -- I've made an upstream issue (see ​https://github.com/karakun/OpenWebStart/issues/557 ). I also made a PR, but they require a CLA, so I had to close said PR. A CLA is (IMO) not worth the headache.

Workaround until OWS fixes this:

• Open OpenWebStart Settings
• JVM Manager
• Settings...
• Modify the Default update server URL: field to https://raw.githubusercontent.com/tsmock/OpenWebStart/master/download-server/resources/jvms.json. You should change it back after it downloads the Java 21 JVM -- I will not be updating the custom jvms.json file.
  • Default server is https://download-openwebstart.com/jvms.json, but clearing out that field will autofill the field with that value.

With that said, it looks like there may be an issue with some OWS code with Java 21:

    at net.sourceforge.jnlp.Launcher$TgThread.run(Launcher.java:660) 
    at net.sourceforge.jnlp.Launcher.access$300(Launcher.java:71) 
    at net.sourceforge.jnlp.Launcher.launchApplication(Launcher.java:373) 
    at net.sourceforge.jnlp.Launcher.createApplication(Launcher.java:584) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.getInstance(JNLPClassLoader.java:468) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.getInstance(JNLPClassLoader.java:495) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.createInstance(JNLPClassLoader.java:423) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.<init>(JNLPClassLoader.java:350) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.initializeResources(JNLPClassLoader.java:709) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.add(JarCertVerifier.java:225) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.verifyJars(JarCertVerifier.java:251) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.verifyJar(JarCertVerifier.java:290) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.verifyJarEntryCerts(JarCertVerifier.java:366) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.checkCertUsage(JarCertVerifier.java:612) 
Caused by: java.lang.NoSuchMethodError: 'java.lang.Boolean sun.security.x509.NetscapeCertTypeExtension.get(java.lang.String)' 
    at net.sourceforge.jnlp.Launcher$TgThread.run(Launcher.java:681) 
Exception in thread "JOSM (development version)" java.lang.RuntimeException: java.lang.NoSuchMethodError: 'java.lang.Boolean sun.security.x509.NetscapeCertTypeExtension.get(java.lang.String)'

It looks like IcedTea-Web (the base for OpenWebStart) needs to make some changes to support Java 21.

[taylor.smock]

Thanks, not sure what to do.

For now, you probably need to use the JAR file. I'll see if I can get the CI to start building JOSM installers with Java 21.

[richlv]

Thank you so much for looking into this, I guess all those issues will end up in other tickets.

Regarding the FIT plugin, it worked great with several files, then I imported a bunch of files at once, and got an exception.
Is this sufficient info to find out what caused it, or should I track down the exact file?

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2023-08-29 13:38:40 +0200 (Tue, 29 Aug 2023)
Revision:18822
Build-Date:2023-08-30 01:30:57
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (18822 en_GB) Mac OS X 13.6
OS Build number: macOS 13.6 (22G120)
Memory Usage: 494 MB / 4096 MB (244 MB allocated, but free)
Java version: 21+35-2513, Oracle Corporation, OpenJDK 64-Bit Server VM
Look and Feel: com.apple.laf.AquaLookAndFeel
Screen: Display 1127230987 1920×1080 (scaling 1.00×1.00) Display 69733382 1680×1050 (scaling 2.00×2.00)
Maximum Screen Size: 1920×1080
Best cursor sizes: 16×16→16×16, 32×32→32×32
System property file.encoding: UTF-8
System property sun.jnu.encoding: UTF-8
Locale info: en_GB
Numbers with default locale: 1234567890 -> 1234567890

Plugins:
+ FIT (36151)
+ HouseNumberTaggingTool (36126)
+ InfoMode (36126)
+ Mapillary (2.2.0)
+ PicLayer (1.0.3)
+ apache-commons (36034)
+ apache-http (35924)
+ buildings_tools (36134)
+ dataimport (36066)
+ ejml (35924)
+ geotools (36068)
+ imagery_offset_db (36126)
+ jackson (36034)
+ jaxb (36118)
+ jna (36126)
+ jts (36004)
+ measurement (36126)
+ opendata (36126)
+ pbf (36126)
+ photo_geotagging (36126)
+ reverter (36126)
+ undelete (36126)
+ utilsplugin2 (36134)

Map paint styles:
- https://josm.openstreetmap.de/josmfile?page=Styles/Coloured_Streets&zip=1

Last errors/warnings:
- 00173.430 E: Handled by bug report queue: java.lang.IllegalArgumentException: Illegal Capacity: -1



=== REPORTED CRASH DATA ===
BugReportExceptionHandler#handleException:
No data collected.

Warning issued by: BugReportExceptionHandler#handleException

=== STACK TRACE ===
Thread: main-worker-0 (34) of main
java.lang.IllegalArgumentException: Illegal Capacity: -1
	at java.base/java.util.ArrayList.<init>(ArrayList.java:160)
	at org.openstreetmap.josm.plugins.fit.lib.FitReader.readNextDefinition(FitReader.java:201)
	at org.openstreetmap.josm.plugins.fit.lib.FitReader.read(FitReader.java:68)
	at org.openstreetmap.josm.plugins.fit.FitPlugin$FitImporter.importData(FitPlugin.java:56)
	at org.openstreetmap.josm.gui.io.importexport.FileImporter.importDataHandleExceptions(FileImporter.java:103)
	at org.openstreetmap.josm.actions.OpenFileAction$OpenFileTask.importData(OpenFileAction.java:401)
	at org.openstreetmap.josm.actions.OpenFileAction$OpenFileTask.realRun(OpenFileAction.java:350)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.doRealRun(PleaseWaitRunnable.java:94)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.run(PleaseWaitRunnable.java:142)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

[taylor.smock]

This most likely occurred due to hitting the end of the file. From InputStream.read documentation:

Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown.

If you can figure out which file caused it and can share it, that would be appreciated. We have two options:

• I'm messing something up when reading the fit file
• The fit file is corrupted in some manner (truncated, a bit flipped, etc.)

[richlv]

Turns out it's one of the provided testcases, 2023-09-10-11-09-20.fit​ .

[richlv]

Noticed another problem with another file, longer route.
If I convert it to GPX with gpsbabel, it loads as GPX and as "markers from..." layers.
If I open FIT directly, there are no markers - but there are lines from (3 out of 4) marker locations to null island.
Attaching FIT, GPX and screenshots of the problem.

[richlv]

Trace, imported from GPX

[richlv]

Trace, imported from FIT

[richlv]

FIT file that results in lines to null island

[richlv]

GPX file where FIT resulted in lines to null island

[taylor.smock]

In 36158/osm:

See #8647: FIT file support

This fixes an issue where there are apparent lines to null island. There were two
reasons for this:

• A point around -1, -1 at a far-distant time point. There are two unknown fields with -1; the "good" data had values >= 0 for both fields (field 13 and field 107).
• Invalid values which were mapped to 180, 180.

[taylor.smock]

In 36159/osm:

See #8647 (dist): FIT file support

[richlv]

Just now noticed that the issue is closed, thought the exception was still an open item.
Updated the plugin, and the affected file now is opened successfully.
With the other file, lines to null island are gone as well.
Thank you so much for this extremely helpful plugin :)