Modify

Opened 11 years ago

Closed 6 months ago

Last modified 5 months ago

#8647 closed enhancement (fixed)

Garmin FIT import

Reported by: pschonmann Owned by: team
Priority: normal Milestone:
Component: Plugin fit Version:
Keywords: import, FIT, Garmin Cc:

Description

Hi,
New garmin devices for bikes, like Edge series arent exporting tracks to GPX, but into FIT. Should be fine import that file directly without converting it in gpsbabel or other utility.

Attachments (11)

2013-05-26-11-02-15.fit (138.4 KB ) - added by mvestola 11 years ago.
Example fit file produced by Garmin Edge 800 during cycling
2013-06-01-16-10-20.gpx (726.4 KB ) - added by mvestola 11 years ago.
Example fit file converted to GPX via Garmin Connect
Fit.java (3.3 KB ) - added by simon04 10 years ago.
2023-09-10-11-09-20.fit.gpx (322.8 KB ) - added by richlv 6 months ago.
FIT testcase 1, Edge 840 produced
2023-09-09-12-00-16.fit.gpx (2.1 MB ) - added by richlv 6 months ago.
FIT testcase 2, Edge 840 produced
2023-09-10-11-09-20.fit (53.2 KB ) - added by richlv 6 months ago.
FIT testcase 1, Edge 840 produced
2023-09-09-12-00-16.fit (337.3 KB ) - added by richlv 6 months ago.
FIT testcase 2, Edge 840 produced
from_gpx.png (26.5 KB ) - added by richlv 6 months ago.
Trace, imported from GPX
from_fit.png (7.9 KB ) - added by richlv 6 months ago.
Trace, imported from FIT
2023-09-10-15-35-22.fit (84.7 KB ) - added by richlv 6 months ago.
FIT file that results in lines to null island
2023-09-10-15-35-22.fit.gpx (521.0 KB ) - added by richlv 6 months ago.
GPX file where FIT resulted in lines to null island

Change History (52)

comment:1 by skyper, 11 years ago

Type: defectenhancement

comment:2 by skyper, 11 years ago

Owner: changed from team to pschonmann
Status: newneedinfo

Is it not possible to change to export gpx ?

What is the advantage of FIT ?

comment:3 by pschonmann, 11 years ago

No option to export/output file to GPX in device.
FIT file has less size than gpx even i record 1s and record HR, CADENCE from ant. Maybe this is some Garmin magic.

I dont know any advantage from gpx, but would be nice have direct import rather do some magic with convert tools thats hard to use for begginers.

comment:4 by Don-vip, 11 years ago

Owner: changed from pschonmann to team
Status: needinfonew

comment:5 by stoecker, 11 years ago

Component: CorePlugin dataimport

Nothing for the core.

comment:7 by Don-vip, 11 years ago

Also, you should attach a .fit sample file if you want this ticket resolved some day, because developers do not have all devices released by Garmin :D

by mvestola, 11 years ago

Attachment: 2013-05-26-11-02-15.fit added

Example fit file produced by Garmin Edge 800 during cycling

by mvestola, 11 years ago

Attachment: 2013-06-01-16-10-20.gpx added

Example fit file converted to GPX via Garmin Connect

comment:8 by mvestola, 11 years ago

I would really need this feature too. I have Garmin Edge 800 which I use in my mountain bike. Edge 800 always outputs the GPS trail using the fit file format. The device has a very precise GPS unit and thus is an excellent device for OSM mapping. I know this is a very popular device with bikers.

However, it is a shame that I can't directly import the GPS trails from my Edge 800 to JOSM because the fit format is not supported by JOSM. But now I need to first convert the fit file to gpx via Garmin Connect website (because the linux version I use in my computer does not have the latest version of gpsbabel which has the fit file support and I don't like building the software from the git sources). I honestly must say that this limits my OSM mapping due to the time-consuming converting between fit and gpx formats.

I attached an example fit file I recorded with my Edge 800. I also attached the same GPS trail in GPX format. I converted the fit file to GPX with garmin connect website. I'm not 100% sure that the files have exactly the same information since the fit file is binary encoded but I assume that the Garmin connect produces the same GPS trail for the GPX file.

comment:9 by simon04, 10 years ago

I obtained a working implementation of a FIT importer based on fit.jar from http://www.thisisant.com/resources/fit. I'm unsure whether the "Flexible and Interoperable Data Transfer (FIT) Protocol License" is compatible with GPL.

by simon04, 10 years ago

Attachment: Fit.java added

comment:10 by simon04, 10 years ago

Summary: Garmin FIT import[Patch] Garmin FIT import

in reply to:  9 comment:11 by bastiK, 10 years ago

Replying to simon04:

I obtained a working implementation of a FIT importer based on fit.jar from http://www.thisisant.com/resources/fit. I'm unsure whether the "Flexible and Interoperable Data Transfer (FIT) Protocol License" is compatible with GPL.

This license is definitely not compatible with GPL. What we could do:

  • find a Java library for FIT under GPL
  • port the GPSBabel code to Java
  • work around the GPL restrictions, e.g.
    • Create a stand alone application (jar file) that can convert FIT to GPX. Make it available on the JOSM website.
    • Let the JOSM-Plugin download this application to the home folder, the first time it is needed. Do not link the jar file, but call it as an external application (something like new ProcessBuilder("java","~/.josm/fit2gpx.jar",fit_filename).start();)

comment:13 by simon04, 8 years ago

Summary: [Patch] Garmin FIT importGarmin FIT import

comment:14 by MaksVasilev, 4 years ago

Open source (GPL2) software Golden Cheetah have native support FIT format:

https://github.com/GoldenCheetah/GoldenCheetah/blob/master/src/FileIO/FitRideFile.cpp

GPX format a deprecated, and may be discontinued.

comment:15 by yourealwaysbe, 4 years ago

I recently implemented a Java FIT to GPX conversion by porting the GPSBabel code instead of using the ANT SDK. It's not widely tested yet, but might be useful. I used the GPL license since that's what GPSBabel uses, and borrowed some tests from MaksVasilev's fit2gpx.

https://gitlab.com/Hague/fitgpxconverter

comment:16 by richlv, 6 months ago

Got an Edge device as a gift, and have a trace for editing, but it's in the fit format...
Any chance to get them supported by JOSM in near future?

comment:17 by taylor.smock, 6 months ago

It looks like they have documented the protocol here: https://developer.garmin.com/fit/protocol/

There isn't a click-through license, so I might be able to do an independent implementation.

comment:18 by richlv, 6 months ago

That would be marvelous. Would there be any benefit from test files?
Definitely happy to help with testing as much as I could.

comment:19 by taylor.smock, 6 months ago

Would there be any benefit from test files?

Yes, if you don't mind them (a) being licensed in a compatible manner (GPLv2+, CC0, CC-BY, CC-BY-SA) and (b) in JOSM source code.

With that said, I don't know when I'll have the bandwidth to do this.

comment:20 by richlv, 6 months ago

Sounds great - attaching two fit files that can be used for testing.
Absolutely understandable that this depends on time availability, will keep an eye on this ticket and try any new development whenever it arrives :)

by richlv, 6 months ago

Attachment: 2023-09-10-11-09-20.fit.gpx added

FIT testcase 1, Edge 840 produced

by richlv, 6 months ago

Attachment: 2023-09-09-12-00-16.fit.gpx added

FIT testcase 2, Edge 840 produced

comment:21 by taylor.smock, 6 months ago

Sounds great - attaching two fit files that can be used for testing.

You attached gpx files, probably converted from fit files.

by richlv, 6 months ago

Attachment: 2023-09-10-11-09-20.fit added

FIT testcase 1, Edge 840 produced

by richlv, 6 months ago

Attachment: 2023-09-09-12-00-16.fit added

FIT testcase 2, Edge 840 produced

comment:22 by richlv, 6 months ago

Ouch, such a fail. Sorry about that.
Uploaded FITs now. Looks like I cannot delete the GPX files - if somebody can do so, please go ahead.

comment:23 by taylor.smock, 6 months ago

Looks like I cannot delete the GPX files

I'm not worried about that. In fact, the gpx files might be useful as a sanity check.

comment:24 by taylor.smock, 6 months ago

Resolution: fixed
Status: newclosed

In 36151/osm:

Fix #8647: Add Garmin FIT parsing (as a plugin)

Documentation/files used for implementation:

comment:25 by taylor.smock, 6 months ago

In 36152/osm:

See #8647 (dist): Add Garmin FIT parsing (as a plugin)

comment:26 by taylor.smock, 6 months ago

Component: Plugin dataimportPlugin fit

comment:27 by taylor.smock, 6 months ago

Notes on the plugin:

  • Requires Java 21 (I probably could have gone with Java 19, but that isn't an "LTS" release)
  • Supports drag-n-drop
  • Does not currently support remote URLs
  • All decoded gps points are part of the same track
Last edited 6 months ago by taylor.smock (previous) (diff)

comment:28 by richlv, 6 months ago

Wow, that's great, thank you so much :)
This finally got me to upgrade Java... but now it doesn't start, as a proxy apparently was configured that cannot be found now ("Creating of proxy failed"). This completely stop the startup.
Will test the new plugin as soon as I can get around this.

comment:29 by richlv, 6 months ago

There seems to be a bit nasty bug in OpenWebStart - if it's set to "system settings" for proxy, system is set to a pac-file, and the host cannot be resolved, startup completely fails.
Found the settings app, manually changed the proxy setting for now.

Got stuck on Java version again - latest from https://openwebstart.com/download/ installed 17.0.7 Azul Systems, and the new plugin doesn't like it (as mentioned, requires 21).
Poking around the settings app, cannot find a way to get 21 there.

comment:30 by richlv, 6 months ago

Downloaded OpenJDK21, added it in OpenWebStart settings, removed the Azul one.
On the next startup it downloads Azul again and uses that.
Perhaps caused by josm.jnlp having this?

<java version="1.8+" vendor="Azul"

comment:31 by taylor.smock, 6 months ago

That might be the case. Azul does have a Java 21 JDK. See https://www.azul.com/downloads/?version=java-21-lts&package=jdk-fx#zulu .

comment:32 by richlv, 6 months ago

Thanks, not sure what to do.
Stick with OpenJDK? If so, how to make OWS work with it?
Stick with Azul? If so, why - and why doesn't jnlp/OWS load 21?

comment:33 by taylor.smock, 6 months ago

Stick with Azul? If so, why - and why doesn't jnlp/OWS load 21?

OWS needs to have the jvms.json updated -- I've made an upstream issue (see https://github.com/karakun/OpenWebStart/issues/557 ). I also made a PR, but they require a CLA, so I had to close said PR. A CLA is (IMO) not worth the headache.

Workaround until OWS fixes this:

  • Open OpenWebStart Settings
  • JVM Manager
  • Settings...
  • Modify the Default update server URL: field to https://raw.githubusercontent.com/tsmock/OpenWebStart/master/download-server/resources/jvms.json. You should change it back after it downloads the Java 21 JVM -- I will not be updating the custom jvms.json file.
    • Default server is https://download-openwebstart.com/jvms.json, but clearing out that field will autofill the field with that value.

With that said, it looks like there may be an issue with some OWS code with Java 21:

    at net.sourceforge.jnlp.Launcher$TgThread.run(Launcher.java:660) 
    at net.sourceforge.jnlp.Launcher.access$300(Launcher.java:71) 
    at net.sourceforge.jnlp.Launcher.launchApplication(Launcher.java:373) 
    at net.sourceforge.jnlp.Launcher.createApplication(Launcher.java:584) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.getInstance(JNLPClassLoader.java:468) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.getInstance(JNLPClassLoader.java:495) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.createInstance(JNLPClassLoader.java:423) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.<init>(JNLPClassLoader.java:350) 
    at net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.initializeResources(JNLPClassLoader.java:709) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.add(JarCertVerifier.java:225) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.verifyJars(JarCertVerifier.java:251) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.verifyJar(JarCertVerifier.java:290) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.verifyJarEntryCerts(JarCertVerifier.java:366) 
    at net.sourceforge.jnlp.tools.JarCertVerifier.checkCertUsage(JarCertVerifier.java:612) 
Caused by: java.lang.NoSuchMethodError: 'java.lang.Boolean sun.security.x509.NetscapeCertTypeExtension.get(java.lang.String)' 
    at net.sourceforge.jnlp.Launcher$TgThread.run(Launcher.java:681) 
Exception in thread "JOSM (development version)" java.lang.RuntimeException: java.lang.NoSuchMethodError: 'java.lang.Boolean sun.security.x509.NetscapeCertTypeExtension.get(java.lang.String)'

It looks like IcedTea-Web (the base for OpenWebStart) needs to make some changes to support Java 21.

comment:34 by taylor.smock, 6 months ago

Thanks, not sure what to do.

For now, you probably need to use the JAR file. I'll see if I can get the CI to start building JOSM installers with Java 21.

comment:35 by richlv, 6 months ago

Thank you so much for looking into this, I guess all those issues will end up in other tickets.

Regarding the FIT plugin, it worked great with several files, then I imported a bunch of files at once, and got an exception.
Is this sufficient info to find out what caused it, or should I track down the exact file?

Relative:URL: ^/trunk
Repository:UUID: 0c6e7542-c601-0410-84e7-c038aed88b3b
Last:Changed Date: 2023-08-29 13:38:40 +0200 (Tue, 29 Aug 2023)
Revision:18822
Build-Date:2023-08-30 01:30:57
URL:https://josm.openstreetmap.de/svn/trunk

Identification: JOSM/1.5 (18822 en_GB) Mac OS X 13.6
OS Build number: macOS 13.6 (22G120)
Memory Usage: 494 MB / 4096 MB (244 MB allocated, but free)
Java version: 21+35-2513, Oracle Corporation, OpenJDK 64-Bit Server VM
Look and Feel: com.apple.laf.AquaLookAndFeel
Screen: Display 1127230987 1920×1080 (scaling 1.00×1.00) Display 69733382 1680×1050 (scaling 2.00×2.00)
Maximum Screen Size: 1920×1080
Best cursor sizes: 16×16→16×16, 32×32→32×32
System property file.encoding: UTF-8
System property sun.jnu.encoding: UTF-8
Locale info: en_GB
Numbers with default locale: 1234567890 -> 1234567890

Plugins:
+ FIT (36151)
+ HouseNumberTaggingTool (36126)
+ InfoMode (36126)
+ Mapillary (2.2.0)
+ PicLayer (1.0.3)
+ apache-commons (36034)
+ apache-http (35924)
+ buildings_tools (36134)
+ dataimport (36066)
+ ejml (35924)
+ geotools (36068)
+ imagery_offset_db (36126)
+ jackson (36034)
+ jaxb (36118)
+ jna (36126)
+ jts (36004)
+ measurement (36126)
+ opendata (36126)
+ pbf (36126)
+ photo_geotagging (36126)
+ reverter (36126)
+ undelete (36126)
+ utilsplugin2 (36134)

Map paint styles:
- https://josm.openstreetmap.de/josmfile?page=Styles/Coloured_Streets&zip=1

Last errors/warnings:
- 00173.430 E: Handled by bug report queue: java.lang.IllegalArgumentException: Illegal Capacity: -1



=== REPORTED CRASH DATA ===
BugReportExceptionHandler#handleException:
No data collected.

Warning issued by: BugReportExceptionHandler#handleException

=== STACK TRACE ===
Thread: main-worker-0 (34) of main
java.lang.IllegalArgumentException: Illegal Capacity: -1
	at java.base/java.util.ArrayList.<init>(ArrayList.java:160)
	at org.openstreetmap.josm.plugins.fit.lib.FitReader.readNextDefinition(FitReader.java:201)
	at org.openstreetmap.josm.plugins.fit.lib.FitReader.read(FitReader.java:68)
	at org.openstreetmap.josm.plugins.fit.FitPlugin$FitImporter.importData(FitPlugin.java:56)
	at org.openstreetmap.josm.gui.io.importexport.FileImporter.importDataHandleExceptions(FileImporter.java:103)
	at org.openstreetmap.josm.actions.OpenFileAction$OpenFileTask.importData(OpenFileAction.java:401)
	at org.openstreetmap.josm.actions.OpenFileAction$OpenFileTask.realRun(OpenFileAction.java:350)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.doRealRun(PleaseWaitRunnable.java:94)
	at org.openstreetmap.josm.gui.PleaseWaitRunnable.run(PleaseWaitRunnable.java:142)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

comment:36 by taylor.smock, 6 months ago

This most likely occurred due to hitting the end of the file. From InputStream.read documentation:

Reads the next byte of data from the input stream. The value byte is returned as an int in the range 0 to 255. If no byte is available because the end of the stream has been reached, the value -1 is returned. This method blocks until input data is available, the end of the stream is detected, or an exception is thrown.

If you can figure out which file caused it and can share it, that would be appreciated. We have two options:

  • I'm messing something up when reading the fit file
  • The fit file is corrupted in some manner (truncated, a bit flipped, etc.)

comment:37 by richlv, 6 months ago

Turns out it's one of the provided testcases, 2023-09-10-11-09-20.fit​ .

comment:38 by richlv, 6 months ago

Noticed another problem with another file, longer route.
If I convert it to GPX with gpsbabel, it loads as GPX and as "markers from..." layers.
If I open FIT directly, there are no markers - but there are lines from (3 out of 4) marker locations to null island.
Attaching FIT, GPX and screenshots of the problem.

by richlv, 6 months ago

Attachment: from_gpx.png added

Trace, imported from GPX

by richlv, 6 months ago

Attachment: from_fit.png added

Trace, imported from FIT

by richlv, 6 months ago

Attachment: 2023-09-10-15-35-22.fit added

FIT file that results in lines to null island

by richlv, 6 months ago

Attachment: 2023-09-10-15-35-22.fit.gpx added

GPX file where FIT resulted in lines to null island

comment:39 by taylor.smock, 6 months ago

In 36158/osm:

See #8647: FIT file support

This fixes an issue where there are apparent lines to null island. There were two
reasons for this:

  • A point around -1, -1 at a far-distant time point. There are two unknown fields with -1; the "good" data had values >= 0 for both fields (field 13 and field 107).
  • Invalid values which were mapped to 180, 180.

comment:40 by taylor.smock, 6 months ago

In 36159/osm:

See #8647 (dist): FIT file support

comment:41 by richlv, 5 months ago

Just now noticed that the issue is closed, thought the exception was still an open item.
Updated the plugin, and the affected file now is opened successfully.
With the other file, lines to null island are gone as well.
Thank you so much for this extremely helpful plugin :)

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain team.
as The resolution will be set.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences .
 
Note: See TracTickets for help on using tickets.