Propose and discuss adding two new data types: 32-bit float and 64-bit int

[jbfaden]

When discussing the file listing specification, Jon and I realized that the complexity of using units to describe files bigger than 2GB (with "KiB" for units, for example) could be avoided if we simply allowed long (64-bit signed integer) data. I've also thought that we ought to have a float type (4-byte real number) as well, since these are often used in CDF and would avoid issues with round-off.

We should discuss the impacts of allowing this data type in HAPI 4.0. I suspect it would be relatively trivial to implement in each data system. Servers needn't implement, so there's no burden on them, and for clients this would be simply detecting new string values and handling these values, which are often put into long and double fields anyway.

[rweigel]

Issue with 64-bit integer: Representation as JSON is allowed, but for interoperability, it should be avoided. https://stackoverflow.com/questions/13502398/json-integers-limit-on-size

[rweigel]

@eelcodoornbos will add to this discussion

[eelcodoornbos]

I've looked into this with the help of Claude Code, to verify the ideas discussed during the telecon against my codebase, the various specs, and the number behaviour in JS.

The verdict for 32-bit float is that it will be very easy to support in my client code, and has real benefits: It would save binary transfer bandwidth (better performance for Timeline Viewer). It also potentially would fix an issue where some of my datasets have fill values of, say, -1e31, which currently come in as -9.999999848243207e+30, and so needs a tolerance when handling it (my code replaces HAPI fill values with NaNs before passing the data on to the rendering libraries). To fix this this, data could be declared as float32, but also the text representation (in JSON/CSV) then needs to be capped at the minimal representative number of significant digits (at most 9) for this representation, i.e. servers should avoid using a float64 string formatter for float32 data. Perhaps worth considering adding to the spec if this feature gets adopted. This would also reduce JSON and CSV bandwidth if implemented this way.

Claude's additional recommendation is to use float32 as the keyword, as simply using "float" (next to "double") would introduce ambiguity: "float" means 32-bit in C and 64-bit in Python/JS.

Supporting Int64 in combination with JSON is more tricky, as suspected. JavaScript's JSON parsing would round any number (not enclosed in quotes) above 2^53-1 without giving a warning/error. JSON parsers in some other languages than JavaScript act the same. Enforcing Int64 to only contain numbers below that threshold and/or transferring the values as strings (digits between quotes) and parsing them as JS BigInt objects would add complexity to the spec, but even more to JS client code. BigInt objects are not supported by visualisation libraries like d3, they are not considered to be a plottable type.

If the only use case is exact byte file sizes then just using "double" for that should be sufficient, as 2^53-1 corresponds to ~9 petabytes. It may not be the prettiest solution from a computer science perspective but works, and is in the end a result of a deliberate limitation in the JSON format and its JS heritage. RFC 8259 §6 explicitly notes that implementations widely use IEEE-754 double and that interoperable implementations should stay within that range. This is not something that is likely to get fixed in a few years.

[rweigel]

Explanation for -1e31 related issue: IHDE-Alliance/ISTP_metadata#10

With compression, how much time savings is there?

[eelcodoornbos]

I had Claude simulate this: With compression, there is most likely no bandwidth savings for binary HAPI requests if the baseline with 64-bit floats doesn't have much information in those additional bits (if the source is a float32 CDF which is auto-converted to float64/double before serving via HAPI). The compression throws away those same useless extra bits, there's no double savings possible. Modest savings of 20-30% are possible in the JSON or CSV output, from the shortening of the float32 string representation compared to an always-at-17-digit double representation, if that would be part of the spec. So bandwidth/time savings are perhaps not the best argument. There's still the exactness/round-off issue though. I'm not too worried about that for my visualisation application, but space agencies and other instrument owners might want to serve the exact same numbers as in their original files.

[jvandegriff]

Here is a ChatGPT pro analysis of the problem and a proposed solution. I read it and it looks pretty good (it took 13 minutes to come up wiht this). It agreed with Eelco's suggestions of using float32 and it suggests an int64 and then we have to place some caveats for use of that data with JSON formats. This could just be part of the spec - if you have int64 numbers, you just won't be able to reliably get them out via JSON.

AI suggestions for how to add long ints and single precision to HAPI

Native 64-bit Integer Support in JavaScript

As of August 2026, there does not appear to be an active TC39 proposal to add a distinct fixed-width int64 or uint64 primitive type to JavaScript. Instead, JavaScript's intended native solution for 64-bit integers is BigInt, which is already part of ECMAScript. The original BigInt proposal explicitly lists interchange with systems that use 64-bit integers as one of its main motivations.

Reference: https://github.com/tc39/proposal-bigint

For example, JavaScript can already represent the full signed 64-bit range exactly:

const min = -9223372036854775808n;
const max =  9223372036854775807n;

console.log(max + 1n);   // 9223372036854775808n

JavaScript also has APIs specifically intended to impose 64-bit semantics on BigInts:

BigInt.asIntN(64, value)   // signed 64-bit
BigInt.asUintN(64, value)  // unsigned 64-bit

And JavaScript has fixed-width storage types for 64-bit integer quantities:

const signed = new BigInt64Array(100);
const unsigned = new BigUint64Array(100);

Those arrays store true signed and unsigned 64-bit integer values, with values exposed to JavaScript as BigInts.

So TC39 effectively chose:

arbitrary-precision BigInt plus 64-bit conversion/storage facilities

rather than separate int64 and uint64 scalar primitives. TC39 discussions indicate that BigInt initially considered something more like Int64, but moved toward the more general BigInt design.

Reference: https://github.com/tc39/notes/blob/main/meetings/2020-02/february-4.md

The interesting development for HAPI is JSON

This has changed somewhat since the traditional warning that "JavaScript can't safely read 64-bit integers from JSON."

The problem is still real with ordinary code:

JSON.parse('{"x":9223372036854775807}')

Historically, x becomes a JavaScript Number, which cannot exactly represent that value. Number has only 53 bits of integer precision.

However, ECMAScript 2026 has added JSON.parse source-text access, a completed Stage-4 TC39 feature. A reviver can now obtain the original digits and convert them directly to BigInt before relying on the rounded Number.

Reference: https://github.com/tc39/proposal-json-parse-with-source

const s = '{"x":9223372036854775807}';

const obj = JSON.parse(s, (key, value, context) => {
    if (key === "x") {
        return BigInt(context.source);
    }
    return value;
});

console.log(obj.x);
// 9223372036854775807n

MDN also documents this technique for recovering large JSON integers without precision loss.

Reference: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse

There is now also JSON.rawJSON(), which allows a BigInt to be emitted as a JSON numeric token using a replacer:

const obj = {
    x: 9223372036854775807n
};

const json = JSON.stringify(obj, (key, value) =>
    typeof value === "bigint"
        ? JSON.rawJSON(value.toString())
        : value
);

console.log(json);
// {"x":9223372036854775807}

So a standards-compliant JSON document can contain the integer as a normal JSON number, and modern JavaScript has mechanisms for preserving it exactly.

What this means for adding int64 to HAPI

This makes me considerably more comfortable with an HAPI int64 type than the old "JavaScript can't handle it" argument would suggest.

Layer	64-bit integer support
JavaScript language	Yes, exactly, via BigInt
JavaScript 64-bit arrays	Yes, BigInt64Array / BigUint64Array
JavaScript fixed scalar int64 type	No, and no major TC39 proposal appears to be pursuing one
JSON syntax	Yes — JSON numbers have no specified 53-bit restriction
Traditional JSON.parse()	Unsafe by default above 2^53 - 1
Modern JSON.parse()	Can preserve exact value using context.source + BigInt
JSON.stringify(BigInt) directly	Still not automatic
Modern serialization	Possible using JSON.rawJSON()

For HAPI specifically, the /info response tells the client what type each parameter has. A HAPI client that sees something like this:

{
    "name": "count",
    "type": "integer64"
}

could know that the corresponding numeric fields in /data need to be converted from their source digits to BigInt, rather than to Number.

Bottom line

I would not wait for JavaScript to acquire an int64 primitive before adding 64-bit integer support to HAPI. BigInt is already the JavaScript language's answer to that problem.

The more important specification question is how HAPI says a JSON numeric value must be parsed when its declared type is int64, particularly for older JavaScript environments and generic JSON parsers.

[eelcodoornbos]

What are the actual use cases for supporting int64 where other types (e.g. float64 or string) are not already "good enough"? Is it wise to add things to the spec if there's no good way they can be used hassle-free via other parts of the spec?
The HAPI spec has already settled from the start on simply using strings for timetags, which is one area where other systems might employ int64. File sizes in bytes can easily use float64. Other identifiers can also use strings.
Would introducing the int64 type, even with caveats in the spec, give implementers a "freedom" that will ultimately reduce compatibility between some HAPI servers and some HAPI clients?

[jbfaden]

One of (my/the) design goals for HAPI is that everything in CDAWeb can be represented precisely in HAPI. Since 32-bit real and 64-bit ints are found in CDF files, then it's worth considering these. I do agree though, that there are fairly simple work-arounds, like using doubles for floats is fine most of the time, and strings could be used to transmit 64-bit ints. (In fact since 64-bit ints are often used for status flags, a hexidecimal representation in a string would probably be more useful anyway.)

Also--we already have string types, which constrain the values in a string. (For example "41° 39' 40'' N") We could have a string type which is a 64-bit integer represented in hexidecimal.

[rweigel]

Python json.loads(), Javascript in current Chrome, and Julia handle. IDL and MATLAB don't. A custom JSON reader would be needed, and a guard would be needed to prevent numbers from being wrapped in IDL or truncated in MATLAB (not tested, based on 1 and 2). Jackon for Java would need mapper.enable(DeserializationFeature.USE_BIG_INTEGER_FOR_INTS); added (based on 3).

[jbfaden]

IDL has 32-bit floats and 64-bit integers. They've always had floats and they added 64-bit integers (LONG64) around 2000. Do you mean the language or the libraries the language uses?