EUC World API for 3rd party apps / plugins

[supercurio]

Hi @Seba!

I'm now starting to make an 3rd party app (or EUC World plugin) that'll control LED light strips according to speed, braking and battery level.
Technically, the LED strips are connected to a SP110E controller with BLE functionality and the app will read the wheel data from EUC World API.

A typed, native Android API would be more more energy-efficient and ergonomic to work with than JSON on a local web server, but it's okay for now.

Here are a couple things missing:

Automatic start and stop of the 3rd party app / plugin following EUC World lifecycle

Unless creating a foreground service (with notification) which runs at all time and checks periodically for the availability of EUC World local HTTP server on the port 8080, there's no good way to be sure to operate when EUC World will be running and connected to a wheel.

Notification of new wheel data available

Since the current API is just a static document generated upon request, there is no way to know when new wheel data will be available.
Depending on which wheel, new data might come at a constant sampling frequency or only sometimes, several times per second.

Although subscribing for data updates via a native API would be a better option, a way to be notified of new data on the Web API would still be desirable in the meantime.

For my app's use case, for now polling the API at high enough frequency is the only way to reduce the latency to detect braking action and reflect that on the LED strips. It'll waste some battery however.

 

Nothing preventing me to work on a first version tho!
I'll be happy to help shaping a robust API for 3rd party apps / plugins integrating with EUC World, as Android dev by trade with 11+ years of experience.

Edited May 11, 2021 by supercurio

[Seba]

12 hours ago, supercurio said:

Unless creating a foreground service (with notification) which runs at all time and checks periodically for the availability of EUC World local HTTP server on the port 8080, there's no good way to be sure to operate when EUC World will be running and connected to a wheel.

I can add broadcasts for most important events, so your app could subscribe to these broadcasts.

12 hours ago, supercurio said:

Since the current API is just a static document generated upon request, there is no way to know when new wheel data will be available.
Depending on which wheel, new data might come at a constant sampling frequency or only sometimes, several times per second.

I'm aware of this limitation, which is why I assumed from the very beginning that the static REST HTTP-based API would be supplemented via WebSockets. I hope to add this functionality in the near future. I just have to deal with more important things first.

[supercurio]

10 hours ago, Seba said:

I can add broadcasts for most important events, so your app could subscribe to these broadcasts.

I'm aware of this limitation, which is why I assumed from the very beginning that the static REST HTTP-based API would be supplemented via WebSockets. I hope to add this functionality in the near future. I just have to deal with more important things first.

Yeah, my app will have a service which could listen to an intent broadcasted.

An intent broadcasted could only tell "there's new data now", that would work.
The intent broadcast could also have extras, as a bundle containing all the data  - possibly with the same keys as the JSON.

I understand, WebSockets for web-apps make sense.

 

In the next few weeks, while I'll make progress on my prototype, I'll also create a repo with proposed API for plugins.
I intend to make that as a sort-of RFC, with the result being free & open source libraries you could use as-is in the closed source EUC World app, as well as other plugin developers in their own app on the other side.

Since you likely have a really long TODO list already and this type of functionality won't be high priority, I guess this approach could be a reasonable way to make progress faster.

[supercurio]

Progress update

I started yesterday evening and right now there is:

LED control:

• On/Off
• Set light/color pattern
• Set Pattern speed
• Set brightness
• Set Brake light

EUC World data source:

• Monitors when EUC World starts (runs at all times, checks 127.0.0.1:8080 every 5s)
• Fetches wheel data from EUC World web API as fast as possible

This is where the first limitations appear

Before parsing the JSON, the resource usage is, on a Google Pixel 2:

• 32% CPU usage from EUC World when not connected to a wheel
• 45% CPU usage from EUC World when connected to a 16X
• 10% CPU usage from my app

Both tops out at roughly 19 requests per second. I imagined it would be a lot more.

Early conclusions

An API which is both push and native will be required for efficient operation.

Push will solve the waste of pulling several times the same unchanged data.
Native, or maybe protobuf over a socket will solve the excessive CPU usage and battery drain.

The current approach will work for an early proof of concept prototype, but will drain the phone's battery in a very short amount of time for features that require low latency like the brake light from LED strips. It is impractical for production.

I'm happy to play with what's possible already with EUC World 2.4.4 tho!

@Seba is there a window for a code merge for a native API in EUC World 2.6?
I can write something shortly that you could use directly, which would address all that.

Cheers,

Edited May 15, 2021 by supercurio

[Seba]

27 minutes ago, supercurio said:

Push will solve the waste of pulling several times the same unchanged data.

As a quick hint - you can limit what parameters are sent in response by adding "filter" parameter to the GET query. Filter is a regex matching WKIs you need. WKI is a parameter identifier, like "vba" is for battery level, "vsp" is speed etc. There is also another optional parameter "attrs" which control which attributes are returned. By default this value equals 1. If you set it to 0, parameter names won't be returned. If you set it to 2, additional attributes will be returned.

[supercurio]

16 minutes ago, Seba said:

As a quick hint - you can limit what parameters are sent in response by adding "filter" parameter to the GET query. Filter is a regex matching WKIs you need. WKI is a parameter identifier, like "vba" is for battery level, "vsp" is speed etc. There is also another optional parameter "attrs" which control which attributes are returned. By default this value equals 1. If you set it to 0, parameter names won't be returned. If you set it to 2, additional attributes will be returned.

Thanks! It works and will help a bit for the early versions.

The requests per second still cap out at the same (about 20 req/s; I guess this one is intentional), but it can reduce the EUC World CPU usage to around 20% from 32% when filtering with filter=vb.*&attrs=0 compared to the previous example.

[Seba]

1 hour ago, supercurio said:

Thanks! It works and will help a bit for the early versions.

The requests per second still cap out at the same (about 20 req/s; I guess this one is intentional), but it can reduce the EUC World CPU usage to around 20% from 32% when filtering with filter=vb.*&attrs=0 compared to the previous example.

5 reqs/s is more than enough, as this is the fastest rate at which data is received from the wheel

[supercurio]

1 minute ago, Seba said:

5 reqs/s is more than enough, as this is the fastest rate at which data is received from the wheel

Good to know! So it's never that high.

Querying at higher frequency is still needed to avoid missing data points and reduce due to lack of synchronisation.

I'm writing the JSON deserialisation at the moment, here's what I guessed:

• "v" is Value
• "t" is Title
• is "p" for Precision?
• is "d" for Decimal value?
• "i" is always false, what is it? 

Essentially, I'm happy with the title and type - the later I'm not sure for each item. Maybe there's more?

[Seba]

28 minutes ago, supercurio said:

Good to know! So it's never that high.

Querying at higher frequency is still needed to avoid missing data points and reduce due to lack of synchronisation.

I'm writing the JSON deserialisation at the moment, here's what I guessed:

• "v" is Value
• "t" is Title
• is "p" for Precision?
• is "d" for Decimal value?
• "i" is always false, what is it? 

Essentially, I'm happy with the title and type - the later I'm not sure for each item. Maybe there's more?

Sure  I always lacked the time to write an API docs. Here it is:

• "t" - title
• "v" - value
• "n" - numeric value (available only when p = 1)
• "p" - value type:
  • 0 -> string
  • 1 -> combined string ("v") and long ("n")
  • 2 -> int
  • 3 -> long
  • 4 -> float
  • 5 -> double
  • 6 -> temperature [°C]
  • 7 -> speed [km/h]
  • 8 -> distance [km]
  • 9 -> per distance (energy consumption, for example) [/km]
  • 10 -> date [UNIX timestamp in milliseconds]
  • 11 -> altitude [m]
• "d" - true if value is valid (value will be invalidated after disconnecting or during loss of connection)
• "a" - age (UNIX timestamp of last update expressed in milliseconds)
• "i" - true if imperial system should be used for display purpose; otherwise false (values are always sent as metric, this setting reflect user preferences set in app)
• "y" - parameter validity period in milliseconds (value will stay valid for "y" milliseconds from last update)
• "tz" - timezone identifier
• "sm" - suffix for metric system
• "si" - suffix for imperial system

Note - all floats are sent with precision limited to three digits after decimal points, so from JSON point of view there is no difference between float and double.

Edited May 15, 2021 by Seba

[supercurio]

Thanks for taking the time @Seba, appreciated 

[supercurio]

Okay! I have a working prototype app.

There's nothing but a test UI, a lot of things are hardcoded but it works 
I implemented the whole EUC World API JSON in the process, although for now it uses only speed.

I'll publish a build on the Play Store this week and release the source after that.

[supercurio]

I published an early release of my EUC Toolkit app here!

 

@Seba: here's the EUC World API implementation in Kotlin:
https://github.com/supercurio/euc-toolkit-app/blob/release-web-v1/app/src/main/java/supercurio/euctoolkit/datasources/eucworld/EucWorldJsonData.kt

I couldn't tell for sure which was the type of some items which were not populated in the API result, but other than that it's pretty complete already

 

[supercurio]

Is it possible to write to EUC World API @Seba?
I'd like to add quick pause/resume recording toggle as widgets and on the lockscreen.

It's the main ergonomic challenge I have with EUC World.
Tapping the notification goes to a different screen, then you need to slide to the tour one, long press the pause/configure.

It's something I do 1-2 times per tour in average up to 4-5. When it's negative temperatures, removing gloves to do so etc.

So yeah, I'd be happy to experiment with alternatives UX to pause/resume recordings 

Edited May 19, 2021 by supercurio

[supercurio]

Congratulations on the EUC World 2.6 release @Seba!

What is new for 3rd party apps using the EUC World Web API?

[Seba]

5 hours ago, supercurio said:

Congratulations on the EUC World 2.6 release @Seba!

What is new for 3rd party apps using the EUC World Web API?

Thanks  2.6 doesn't contain any changes in API, but next release will extend API by command & control features.

[Rehab1]

I can’t say it enough. Amazing contributions in both dedication and knowledge.  

Edited May 26, 2021 by Rehab1

[supercurio]

3 hours ago, Seba said:

Thanks  2.6 doesn't contain any changes in API, but next release will extend API by command & control features.

Very cool!

Then I will soon start to work on a native API part for apps, using my existing plugin app as source like, showing an example of how it could be implemented:

• Inside EUC World
• In a third-party app

I've not worked on that kind before, but the idea is that third party apps could add a dependency to a library in their gradle config and get immediate access to wheel data when available, without having to implement anything special and in a very energy-efficient, low-latency way.

 

I haven't set a license for my app yet and subsequent work. Which one would work for you @Seba?

Edited May 26, 2021 by supercurio

[supercurio]

I started looking at native solutions would be suitable for an EUC World API, and fortunately there seem to be several good ones.

In my short list for evaluation at the moment:

• AIDL or Messenger for IPC
• Parcelable or Protobuf for object serialisation

Question @Seba: do the wheels send via BLE all the vehicle state in a single packet containing everything periodically at fixed interval, is it a constant stream coming of data little by little over an emulated serial port, or can new single data point arrive at any time via notifications over multiple services and characteristic?

[Seba]

14 hours ago, supercurio said:

I haven't set a license for my app yet and subsequent work. Which one would work for you @Seba?

I think that MPL would be great here.

1 hour ago, supercurio said:

Question @Seba: do the wheels send via BLE all the vehicle state in a single packet containing everything periodically at fixed interval, is it a constant stream coming of data little by little over an emulated serial port, or can new single data point arrive at any time via notifications over multiple services and characteristic?

There is no rule. Some wheels communicate on "request-answer" basis. Some broadcasts data and requests are used to control the wheel. Refresh rate isn't fixed, also certain values are sent in different moment of time. Anywaym EUC World can fire an event each time after entire set or parameters has been refreshed.

[supercurio]

5 hours ago, Seba said:

I think that MPL would be great here.

Sounds good.
I wasn't considering GPL given the intent, and MPL seem to strike a nice balance. Nice.

5 hours ago, Seba said:

There is no rule. Some wheels communicate on "request-answer" basis. Some broadcasts data and requests are used to control the wheel. Refresh rate isn't fixed, also certain values are sent in different moment of time. Anyway EUC World can fire an event each time after entire set or parameters has been refreshed.

Perfect! exactly what I needed to know to orient the API design in the right direction.

One of my targets is to not introduce latency, hopefully to allow cool mods like laser or custom piezo alarms, brake lights controlled over BLE.
The sort for which the absence of added delay could make the difference between impractical and usable.

[supercurio]

@Seba, do you use string keys like ccu, vedt, vil for wheel signals inside the EUC world app or these exist only as output for the Web API?

Also, for elements like battery of voltage which also have min/max associated, do you store them internally as different, unique signals or as an object with the main value and its min/max?

I'm thinking about how to make build what will give the least amount of conversion or boilerplate code for you and within the lib & API.

[Jonah Hax]

Have anyone ever made a working UI for the API so that I can watch recharging status on my PC, since my bluetooth can't reach to my wheel from my PC.. So now I use an old phone connected to my wheel in the other end of the house, then connect to the EUC World web server on my PC, but I need a pretty UI to watch :-D

 

[neo.exe]

Hi @Seba, I'm trying to send the wheel data (safety margin or PWM) to a bluetooth device via an intermediary app.
Apart from the webserver API, do you have any broadcast intent that sends out the data? https://developer.android.com/guide/components/broadcasts

[Seba]

On 11/15/2023 at 1:47 AM, neo.exe said:

Hi @Seba, I'm trying to send the wheel data (safety margin or PWM) to a bluetooth device via an intermediary app.
Apart from the webserver API, do you have any broadcast intent that sends out the data? https://developer.android.com/guide/components/broadcasts

No, as intents doesn't give the flexibility HTTP server does. But in addition, EUC World will soon also include BLE server in addition to HTTP server, so maybe this will be viable solution for you?

[Seba]

I'm now extending Web server to include charging controller API and also user interface for easy remote monitoring and control of charging.