Signal and noise

Storage details

The files are being stored on Amazon's S3 cloud-storage platform. For the first month, standard S3 is used, so the data is available immediately upon request (with request latency of less than one second). After one month, the files are migrated to S3 Glacier Deep Archive, which has much lower storage cost but retrieval latency of up to 48 hours. All the files are still accessible, but you'll have to wait up to 48 hours to get a copy, so it's best to do any processing (such as observable estimation, cross-correlation among stations, signal-health metrics, etc.) during the first month.

The S3 bucket is "s3://gnss-recordings-pmonta", and its region is us-east-1 (Northern Virginia). If you want to do computation with these recordings, it would be cheapest to use EC2 compute resources in that region, since data transfer between S3 and EC2 is then free. If you transfer the files to other AWS regions, or download them over the Internet to your own storage and computation, then transfer costs are imposed.

If you upload GNSS waveforms to AWS, please also use us-east-1 if possible.

Speaking of costs, the S3 bucket is marked "public" and "requester pays". This is so I don't have to foot the bill for possibly voluminous worldwide downloads of these files (I do have to pay for the storage costs though). So downloading these files requires that you have an AWS account. I'd prefer to not have this speed bump, but I don't see any way around it. Sorry. Maybe at some point AWS can make it part of their "free public data set" offering, in which case anonymous access would be free I think. Note that there is a free tier for AWS, which, after signup, allows a limited amount of storage, computation, and data transfer per month.

A shallow hierarchy is used inside the s3://gnss-recordings-pmonta bucket, of the form <year>/<doy>/filename, where <doy> is the ordinal day within the year ranging from 1 to 366. All dates and times are in GPS time, that is, TAI minus 19 seconds.

2020/
  029/
    ...
    PAOC00USA_2020029103500.json
    PAOC00USA_2020029103500_L1.iq.bz2
    ...
    SBTH00USA_2020029103500.json
    SBTH00USA_2020029103500_L1.iq.bz2
    ...

To reduce the friction of getting started with these files, I have a small subset of them available here on a public basis with no need for AWS credentials:

http://gnss-recordings-pmonta-sample.s3-website-us-east-1.amazonaws.com/index.html

This document is an index to the available files: two sets of recordings, seven hours apart, from the two sites. The sample files can be downloaded with a web browser using the provided links, or alternatively with wget as follows:

wget http://gnss-recordings-pmonta-sample.s3-website-us-east-1.amazonaws.com/PAOC00USA_2020029103500.json
wget http://gnss-recordings-pmonta-sample.s3-website-us-east-1.amazonaws.com/PAOC00USA_2020029103500_L1.iq.bz2
...

File format and naming

Each recording consists of a small amount of metadata and sample streams for the various RF bands. There are many choices for packaging them: HDF5, or the recent ION standard for GNSS waveform files [1], or the various RF/IF formats from radio astronomy. After reading this interesting essay about HDF5, archivalness, and software transparency [2], I opted for flat binary files compressed with a well-known compressor (bzip2), together with metadata in JSON. I don't claim it's the best choice, but at least it is easy to understand and can be easily translated into any of these other formats.

Prior to compression, the sample streams consist of 8-bit signed integers, alternating between in-phase and quadrature part (or real and imaginary part): i,q,i,q, etc. This is a common format for SDR sample streams. 8 bits is ordinarily enough precision for GNSS signals; in fact my recordings are 2-bit (4-level), with signal values -3,-1,1,3. (A signal value of 0 is used for missing data resulting from a dropped packet during recording, which is rare.) Higher-precision data, such as 3-bit or 4-bit, would fit seamlessly into this scheme. The compression reduces the file size to approximately the entropy of the underlying source.

File names roughly follow the RINEX3 convention:

• a 9-character site string containing a 4-character site ID, 1-digit marker number, 1-digit receiver number, and 3-character country code</li>
• Date and time (GPS time)</li>
• Suffix, either .json for JSON metadata or <band>.iq.bz2 for IQ sample data</li>

A full set of four files for a given site and time looks like this:

PAOC00USA_2020029103500.json
PAOC00USA_2020029103500_L1.iq.bz2
PAOC00USA_2020029103500_L2.iq.bz2
PAOC00USA_2020029103500_L5.iq.bz2

This scheme results in many small files (about 1000 per site per day), but has the advantage that the user can request just the data desired.

Metadata

I chose metadata fields mostly inspired by RINEX3. In fact one goal is to be able to construct a single-epoch RINEX3 file from each file-set, and for that one needs observables from all the systems of interest and enough RINEX3 metadata to fill in the blanks.

Here is an example JSON file:

{
    "antenna": {
        "serial_number": "xxxx",
        "type": "Tallysman TW3972"
    },
    "approx_position": [
        -2701201.6,
        -4291624.4,
        3855647.9
    ],
    "bands": [
        {
            "center_freq": [
                5797,
                256
            ],
            "filename": "PAOC00USA_2020029103500_L1.iq.bz2",
            "name": "L1",
            "receiver_channel": 1
        },
        {
            "center_freq": [
                4491,
                256
            ],
            "filename": "PAOC00USA_2020029103500_L2.iq.bz2",
            "name": "L2",
            "receiver_channel": 2
        },
        {
            "center_freq": [
                4359,
                256
            ],
            "filename": "PAOC00USA_2020029103500_L5.iq.bz2",
            "name": "L5",
            "receiver_channel": 3
        }
    ],
    "marker_to_ARP": [
        0,
        0,
        0
    ],
    "observer": {
        "name": "Peter Monta"
    },
    "receiver": {
        "serial_number": "70:B3:D5:F7:90:09",
        "software_version": "2",
        "type": "GNSS Firehose"
    },
    "sample_rate": 69984000,
    "site": {
        "country": "USA",
        "marker_number": 0,
        "name": "PAOC",
        "receiver_number": 0
    },
    "time_duration": "0.2",
    "time_start": "1580294099.9"
}

The fields are mostly self-explanatory (when read together with the RINEX3 spec) except for the "bands" list. For each band, a center frequency is given as a ratio of integers P/Q, which is to be multiplied by the sample rate. This represents the coherence between the sample rate and each channel's downconverter local oscillator. If, for example, the user wanted to construct a single wideband signal from multiple overlapping subbands, then the exact ratios would allow the frequency difference to be set in stone, i.e., not estimated. Only the relative phases would need to be estimated. Those phases are fixed because the various PLLs in the receiver frequency chain never go out of lock. (What, never? Well, hardly ever.)

Licensing

I'm not an expert on copyright, but my intent is to provide these files to be used freely by anyone, roughly according to the Creative Commons license CC BY-SA 4.0. If the attribution or derivative portions of this license turn out to be unwieldy, especially if others contribute similar data, then it might be revised. I don't know if entities like IGS or IERS have a formal license for their data, but if so, that might be a good model.

Applications

Possible applications, exclusive of the usual ones involving observables, positioning, etc.:

• satellite health monitoring
• receiver development and testing
• observables obtainable only from waveforms, e.g. cross-correlation of GPS M code, Galileo PRS, etc.
• cooperative estimation of things like W bits or M bits (requires many stations)
• archival "history" of GNSS

Using the current upload schedule, the average data rate per station is about 1 Mbit/s, with a duty cycle of either 0.07% or 0.13% depending on the band. It is not currently feasible to provide megabit-class uplinks from GNSS stations in the middle of nowhere, although the large LEO constellations under construction might change that in the near future. Continuous waveforms would be even better [3], but currently require deep pockets for the storage and bandwidth. A reasonable path might be a gradual increase in duty cycle.

Disclaimer

I can't make any guarantees about data uploads or data availability. "Best effort."

References